{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Graph Neural Network Topic Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following we will focus on building a model for topic classification based on a Graph Neural Network approach.\n",
    "\n",
    "In particular in the following we will show you how to:\n",
    "\n",
    "* Create a TF-IDF representation of the corpus, that will be used as node features in the Graph Neural Network model \n",
    "* Build, train a Graph Neural Network model and identify the best threshold for classifying documents \n",
    "* Test the performance of the model in a out-of-sample tests, following a truly inductive approach \n",
    "\n",
    "**NOTE: This Notebook can only be run after the 01_nlp_graph_creation notebook, as some of the results computed in the first notebook will be here reused.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = pd.read_pickle(\"corpus.p\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>clean_text</th>\n",
       "      <th>parsed</th>\n",
       "      <th>language</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>test/14826</th>\n",
       "      <td>[trade]</td>\n",
       "      <td>ASIAN EXPORTERS FEAR DAMAGE FROM U.S.-JAPAN RI...</td>\n",
       "      <td>(ASIAN, EXPORTERS, FEAR, DAMAGE, FROM, U.S.-JA...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14828</th>\n",
       "      <td>[grain]</td>\n",
       "      <td>CHINA DAILY SAYS VERMIN EAT 7-12 PCT GRAIN STO...</td>\n",
       "      <td>(CHINA, DAILY, SAYS, VERMIN, EAT, 7, -, 12, PC...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14829</th>\n",
       "      <td>[crude, nat-gas]</td>\n",
       "      <td>JAPAN TO REVISE LONG-TERM ENERGY DEMAND DOWNWA...</td>\n",
       "      <td>(JAPAN, TO, REVISE, LONG, -, TERM, ENERGY, DEM...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14832</th>\n",
       "      <td>[corn, grain, rice, rubber, sugar, tin, trade]</td>\n",
       "      <td>THAI TRADE DEFICIT WIDENS IN FIRST QUARTER  Th...</td>\n",
       "      <td>(THAI, TRADE, DEFICIT, WIDENS, IN, FIRST, QUAR...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14833</th>\n",
       "      <td>[palm-oil, veg-oil]</td>\n",
       "      <td>INDONESIA SEES CPO PRICE RISING SHARPLY  Indon...</td>\n",
       "      <td>(INDONESIA, SEES, CPO, PRICE, RISING, SHARPLY,...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     label  \\\n",
       "id                                                           \n",
       "test/14826                                         [trade]   \n",
       "test/14828                                         [grain]   \n",
       "test/14829                                [crude, nat-gas]   \n",
       "test/14832  [corn, grain, rice, rubber, sugar, tin, trade]   \n",
       "test/14833                             [palm-oil, veg-oil]   \n",
       "\n",
       "                                                   clean_text  \\\n",
       "id                                                              \n",
       "test/14826  ASIAN EXPORTERS FEAR DAMAGE FROM U.S.-JAPAN RI...   \n",
       "test/14828  CHINA DAILY SAYS VERMIN EAT 7-12 PCT GRAIN STO...   \n",
       "test/14829  JAPAN TO REVISE LONG-TERM ENERGY DEMAND DOWNWA...   \n",
       "test/14832  THAI TRADE DEFICIT WIDENS IN FIRST QUARTER  Th...   \n",
       "test/14833  INDONESIA SEES CPO PRICE RISING SHARPLY  Indon...   \n",
       "\n",
       "                                                       parsed language  \n",
       "id                                                                      \n",
       "test/14826  (ASIAN, EXPORTERS, FEAR, DAMAGE, FROM, U.S.-JA...       en  \n",
       "test/14828  (CHINA, DAILY, SAYS, VERMIN, EAT, 7, -, 12, PC...       en  \n",
       "test/14829  (JAPAN, TO, REVISE, LONG, -, TERM, ENERGY, DEM...       en  \n",
       "test/14832  (THAI, TRADE, DEFICIT, WIDENS, IN, FIRST, QUAR...       en  \n",
       "test/14833  (INDONESIA, SEES, CPO, PRICE, RISING, SHARPLY,...       en  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "topics = Counter([label for document_labels in corpus[\"label\"] for label in document_labels]).most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('earn', 3964),\n",
       " ('acq', 2369),\n",
       " ('money-fx', 717),\n",
       " ('grain', 582),\n",
       " ('crude', 578),\n",
       " ('trade', 485),\n",
       " ('interest', 478),\n",
       " ('ship', 286),\n",
       " ('wheat', 283),\n",
       " ('corn', 237)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "topicsList = [topic[0] for topic in topics]\n",
    "topicsSet = set(topicsList)\n",
    "dataset = corpus[corpus[\"label\"].apply(lambda x: len(topicsSet.intersection(x))>0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_labels(corpus, topicsList=topicsList):\n",
    "    return corpus[\"label\"].apply(\n",
    "        lambda labels: pd.Series({label: 1 for label in labels}).reindex(topicsList).fillna(0)\n",
    "    )[topicsList]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = get_labels(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>earn</th>\n",
       "      <th>acq</th>\n",
       "      <th>money-fx</th>\n",
       "      <th>grain</th>\n",
       "      <th>crude</th>\n",
       "      <th>trade</th>\n",
       "      <th>interest</th>\n",
       "      <th>ship</th>\n",
       "      <th>wheat</th>\n",
       "      <th>corn</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>test/14826</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14828</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14829</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14832</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/14839</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            earn  acq  money-fx  grain  crude  trade  interest  ship  wheat  \\\n",
       "id                                                                            \n",
       "test/14826   0.0  0.0       0.0    0.0    0.0    1.0       0.0   0.0    0.0   \n",
       "test/14828   0.0  0.0       0.0    1.0    0.0    0.0       0.0   0.0    0.0   \n",
       "test/14829   0.0  0.0       0.0    0.0    1.0    0.0       0.0   0.0    0.0   \n",
       "test/14832   0.0  0.0       0.0    1.0    0.0    1.0       0.0   0.0    0.0   \n",
       "test/14839   0.0  0.0       0.0    0.0    0.0    0.0       0.0   1.0    0.0   \n",
       "\n",
       "            corn  \n",
       "id                \n",
       "test/14826   0.0  \n",
       "test/14828   0.0  \n",
       "test/14829   0.0  \n",
       "test/14832   1.0  \n",
       "test/14839   0.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_features(corpus):\n",
    "    return corpus[\"parsed\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_features_and_labels(corpus):\n",
    "    return get_features(corpus), get_labels(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_test_split(corpus):\n",
    "    train_idx = [idx for idx in corpus.index if \"training/\" in idx]\n",
    "    test_idx = [idx for idx in corpus.index if \"test/\" in idx]\n",
    "    return corpus.loc[train_idx], corpus.loc[test_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = train_test_split(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_spacy_tokenizer(pos_filter=[\"NOUN\", \"VERB\", \"PROPN\"]):\n",
    "    def tokenizer(doc):\n",
    "        return [token.lemma_ for token in doc if (pos_filter is None) or (token.pos_ in pos_filter)] \n",
    "    return tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "cntVectorizer = TfidfVectorizer(\n",
    "    analyzer=my_spacy_tokenizer(),\n",
    "    max_df = 0.25, min_df = 2, max_features = 10000\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainFeatures, _ = get_features_and_labels(train)\n",
    "testFeatures, _ = get_features_and_labels(test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainedTransformed = cntVectorizer.fit_transform(trainFeatures)\n",
    "testTransformed = cntVectorizer.transform(testFeatures)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = pd.concat([\n",
    "    pd.DataFrame.sparse.from_spmatrix(trainedTransformed, index=trainFeatures.index), \n",
    "    pd.DataFrame.sparse.from_spmatrix(testTransformed, index=testFeatures.index)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9034, 10000)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the Graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import stellargraph as sg\n",
    "from stellargraph import StellarGraph, IndexedArray\n",
    "from stellargraph.mapper import GraphSAGENodeGenerator\n",
    "from stellargraph.layer import GraphSAGE\n",
    "\n",
    "from tensorflow.keras import layers, optimizers, losses, metrics, Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "edges = pd.read_pickle(\"bipartiteEdges.p\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "entityTypes = {entity: ith for ith, entity in enumerate(edges[\"type\"].unique())}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'keywords': 0, 'GPE': 1, 'ORG': 2, 'PERSON': 3}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entityTypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "documentFeatures = features.loc[set(corpus.index).intersection(features.index)] #.assign(document=1, entity=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>9990</th>\n",
       "      <th>9991</th>\n",
       "      <th>9992</th>\n",
       "      <th>9993</th>\n",
       "      <th>9994</th>\n",
       "      <th>9995</th>\n",
       "      <th>9996</th>\n",
       "      <th>9997</th>\n",
       "      <th>9998</th>\n",
       "      <th>9999</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>training/9238</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/15296</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/15287</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>training/5938</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/21465</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 10000 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               0     1     2     3     4     5     6     7     8     9     \\\n",
       "id                                                                          \n",
       "training/9238   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "test/15296      0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "test/15287      0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "training/5938   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "test/21465      0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "\n",
       "               ...  9990  9991  9992  9993  9994  9995  9996  9997  9998  9999  \n",
       "id             ...                                                              \n",
       "training/9238  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "test/15296     ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "test/15287     ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "training/5938  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "test/21465     ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "\n",
       "[5 rows x 10000 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documentFeatures.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "entities = edges.groupby([\"target\", \"type\"])[\"source\"].count().groupby(level=0).apply(\n",
    "    lambda s: s.droplevel(0).reindex(entityTypes.keys()).fillna(0)\n",
    ").unstack(level=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "entityFeatures = (entities.T / entities.sum(axis=1)).T.assign(document=0, entity=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = {\"entity\": entityFeatures, \n",
    "         \"document\": documentFeatures}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "stellarGraph = StellarGraph(nodes, \n",
    "                            edges[edges[\"source\"].isin(documentFeatures.index)], \n",
    "                            edge_type_column=\"type\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 23998, Edges: 86849\n",
      "\n",
      " Node types:\n",
      "  entity: [14964]\n",
      "    Features: float32 vector, length 6\n",
      "    Edge types: entity-GPE->document, entity-ORG->document, entity-PERSON->document, entity-keywords->document\n",
      "  document: [9034]\n",
      "    Features: float32 vector, length 10000\n",
      "    Edge types: document-GPE->entity, document-ORG->entity, document-PERSON->entity, document-keywords->entity\n",
      "\n",
      " Edge types:\n",
      "    document-keywords->entity: [78838]\n",
      "        Weights: range=[0.0827011, 1], mean=0.258464, std=0.0898612\n",
      "        Features: none\n",
      "    document-ORG->entity: [4129]\n",
      "        Weights: range=[2, 22], mean=3.24122, std=2.30508\n",
      "        Features: none\n",
      "    document-GPE->entity: [2943]\n",
      "        Weights: range=[2, 25], mean=3.25926, std=2.07008\n",
      "        Features: none\n",
      "    document-PERSON->entity: [939]\n",
      "        Weights: range=[2, 14], mean=2.97444, std=1.65956\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(stellarGraph.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stellargraph.data import EdgeSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitter = EdgeSplitter(stellarGraph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "** Sampled 17369 positive and 17369 negative edges. **\n"
     ]
    }
   ],
   "source": [
    "graphTest, samplesTest, labelsTest = splitter.train_test_split(p=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 23998, Edges: 86849\n",
      "\n",
      " Node types:\n",
      "  entity: [14964]\n",
      "    Features: float32 vector, length 6\n",
      "    Edge types: entity-GPE->document, entity-ORG->document, entity-PERSON->document, entity-keywords->document\n",
      "  document: [9034]\n",
      "    Features: float32 vector, length 10000\n",
      "    Edge types: document-GPE->entity, document-ORG->entity, document-PERSON->entity, document-keywords->entity\n",
      "\n",
      " Edge types:\n",
      "    document-keywords->entity: [78838]\n",
      "        Weights: range=[0.0827011, 1], mean=0.258464, std=0.0898612\n",
      "        Features: none\n",
      "    document-ORG->entity: [4129]\n",
      "        Weights: range=[2, 22], mean=3.24122, std=2.30508\n",
      "        Features: none\n",
      "    document-GPE->entity: [2943]\n",
      "        Weights: range=[2, 25], mean=3.25926, std=2.07008\n",
      "        Features: none\n",
      "    document-PERSON->entity: [939]\n",
      "        Weights: range=[2, 14], mean=2.97444, std=1.65956\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(stellarGraph.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 23998, Edges: 69480\n",
      "\n",
      " Node types:\n",
      "  entity: [14964]\n",
      "    Features: float32 vector, length 6\n",
      "    Edge types: entity-GPE->document, entity-ORG->document, entity-PERSON->document, entity-keywords->document\n",
      "  document: [9034]\n",
      "    Features: float32 vector, length 10000\n",
      "    Edge types: document-GPE->entity, document-ORG->entity, document-PERSON->entity, document-keywords->entity\n",
      "\n",
      " Edge types:\n",
      "    document-keywords->entity: [63057]\n",
      "        Weights: range=[0.0827011, 1], mean=0.258427, std=0.0899773\n",
      "        Features: none\n",
      "    document-ORG->entity: [3296]\n",
      "        Weights: range=[2, 22], mean=3.21572, std=2.2592\n",
      "        Features: none\n",
      "    document-GPE->entity: [2360]\n",
      "        Weights: range=[2, 19], mean=3.24237, std=2.01535\n",
      "        Features: none\n",
      "    document-PERSON->entity: [767]\n",
      "        Weights: range=[2, 14], mean=3, std=1.69163\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(graphTest.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Topic Classification Model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by splitting the data into train, validation and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = labels.reindex(documentFeatures.index).fillna(0)\n",
    "#documentFeatures.drop([\"entity\", \"document\"], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>earn</th>\n",
       "      <th>acq</th>\n",
       "      <th>money-fx</th>\n",
       "      <th>grain</th>\n",
       "      <th>crude</th>\n",
       "      <th>trade</th>\n",
       "      <th>interest</th>\n",
       "      <th>ship</th>\n",
       "      <th>wheat</th>\n",
       "      <th>corn</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>test/16678</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test/15913</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>training/12032</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>training/8366</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>training/10454</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                earn  acq  money-fx  grain  crude  trade  interest  ship  \\\n",
       "id                                                                         \n",
       "test/16678       1.0  0.0       0.0    0.0    0.0    0.0       0.0   0.0   \n",
       "test/15913       1.0  0.0       0.0    0.0    0.0    0.0       0.0   0.0   \n",
       "training/12032   0.0  1.0       0.0    0.0    0.0    0.0       0.0   0.0   \n",
       "training/8366    1.0  0.0       0.0    0.0    0.0    0.0       0.0   0.0   \n",
       "training/10454   0.0  1.0       0.0    0.0    0.0    0.0       0.0   0.0   \n",
       "\n",
       "                wheat  corn  \n",
       "id                           \n",
       "test/16678        0.0   0.0  \n",
       "test/15913        0.0   0.0  \n",
       "training/12032    0.0   0.0  \n",
       "training/8366     0.0   0.0  \n",
       "training/10454    0.0   0.0  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targets.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_test_split(corpus):\n",
    "    graphIndex = [index for index in corpus.index]\n",
    "    \n",
    "    train_idx = [idx for idx in graphIndex if \"training/\" in idx]\n",
    "    test_idx = [idx for idx in graphIndex if \"test/\" in idx]\n",
    "    return corpus.loc[train_idx], corpus.loc[test_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "sampled, hold_out = train_test_split(targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "allNeighbors = np.unique([n for node in sampled.index for n in stellarGraph.neighbors(node)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "subgraph = stellarGraph.subgraph(set(sampled.index).union(allNeighbors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 16927, Edges: 62454\n",
      "\n",
      " Node types:\n",
      "  entity: [10438]\n",
      "    Features: float32 vector, length 6\n",
      "    Edge types: entity-GPE->document, entity-ORG->document, entity-PERSON->document, entity-keywords->document\n",
      "  document: [6489]\n",
      "    Features: float32 vector, length 10000\n",
      "    Edge types: document-GPE->entity, document-ORG->entity, document-PERSON->entity, document-keywords->entity\n",
      "\n",
      " Edge types:\n",
      "    document-keywords->entity: [56647]\n",
      "        Weights: range=[0.0918226, 1], mean=0.25739, std=0.0888008\n",
      "        Features: none\n",
      "    document-ORG->entity: [3032]\n",
      "        Weights: range=[2, 22], mean=3.20877, std=2.21143\n",
      "        Features: none\n",
      "    document-GPE->entity: [2104]\n",
      "        Weights: range=[2, 25], mean=3.25808, std=2.08119\n",
      "        Features: none\n",
      "    document-PERSON->entity: [671]\n",
      "        Weights: range=[2, 14], mean=2.97615, std=1.66958\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(subgraph.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, leftOut = train_test_split(\n",
    "    sampled,\n",
    "    train_size=0.1,\n",
    "    test_size=None,\n",
    "    random_state=42,\n",
    ")\n",
    "\n",
    "validation, test = train_test_split(\n",
    "    leftOut, train_size=0.2, test_size=None, random_state=100,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation = validation[validation.sum(axis=1) > 0]\n",
    "test = test[test.sum(axis=1) > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation: (1168, 10)\n",
      "Test: (4673, 10)\n"
     ]
    }
   ],
   "source": [
    "print(f\"Validation: {validation.shape}\")\n",
    "print(f\"Test: {test.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start  by creating the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "num_samples = [10, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stellargraph.mapper import HinSAGENodeGenerator\n",
    "\n",
    "generator = HinSAGENodeGenerator(subgraph, batch_size, num_samples, head_node_type=\"document\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stellargraph.layer import HinSAGE\n",
    "\n",
    "graphsage_model = HinSAGE(\n",
    "    layer_sizes=[32, 32], generator=generator, bias=True, dropout=0.5,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_inp, x_out = graphsage_model.in_out_tensors()\n",
    "prediction = layers.Dense(units=train.shape[1], activation=\"sigmoid\")(x_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([None, 10])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(inputs=x_inp, outputs=prediction)\n",
    "model.compile(\n",
    "    optimizer=optimizers.Adam(lr=0.005),\n",
    "    loss=losses.binary_crossentropy,\n",
    "    metrics=[\"acc\"],\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now train the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_gen = generator.flow(train.index, train, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_gen = generator.flow(validation.index, validation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "13/13 [==============================] - 215s 17s/step - loss: 0.6139 - acc: 0.1365 - val_loss: 0.4780 - val_acc: 0.4401\n",
      "Epoch 2/50\n",
      "13/13 [==============================] - 169s 13s/step - loss: 0.4675 - acc: 0.4323 - val_loss: 0.4001 - val_acc: 0.4401\n",
      "Epoch 3/50\n",
      "13/13 [==============================] - 162s 13s/step - loss: 0.3973 - acc: 0.4319 - val_loss: 0.3486 - val_acc: 0.4401\n",
      "Epoch 4/50\n",
      "13/13 [==============================] - 153s 12s/step - loss: 0.3447 - acc: 0.4604 - val_loss: 0.3124 - val_acc: 0.4401\n",
      "Epoch 5/50\n",
      "13/13 [==============================] - 144s 11s/step - loss: 0.3090 - acc: 0.4997 - val_loss: 0.2853 - val_acc: 0.4932\n",
      "Epoch 6/50\n",
      "13/13 [==============================] - 159s 13s/step - loss: 0.2886 - acc: 0.5484 - val_loss: 0.2639 - val_acc: 0.6045\n",
      "Epoch 7/50\n",
      "13/13 [==============================] - 187s 15s/step - loss: 0.2612 - acc: 0.6354 - val_loss: 0.2453 - val_acc: 0.6387\n",
      "Epoch 8/50\n",
      "13/13 [==============================] - 203s 16s/step - loss: 0.2509 - acc: 0.6294 - val_loss: 0.2307 - val_acc: 0.6404\n",
      "Epoch 9/50\n",
      "13/13 [==============================] - 178s 14s/step - loss: 0.2370 - acc: 0.6489 - val_loss: 0.2160 - val_acc: 0.6789\n",
      "Epoch 10/50\n",
      "13/13 [==============================] - 190s 15s/step - loss: 0.2155 - acc: 0.6836 - val_loss: 0.2046 - val_acc: 0.7029\n",
      "Epoch 11/50\n",
      "13/13 [==============================] - 172s 14s/step - loss: 0.2047 - acc: 0.7310 - val_loss: 0.1938 - val_acc: 0.7260\n",
      "Epoch 12/50\n",
      "13/13 [==============================] - 145s 12s/step - loss: 0.2009 - acc: 0.7208 - val_loss: 0.1846 - val_acc: 0.7509\n",
      "Epoch 13/50\n",
      "13/13 [==============================] - 167s 13s/step - loss: 0.1834 - acc: 0.7843 - val_loss: 0.1755 - val_acc: 0.7860\n",
      "Epoch 14/50\n",
      "13/13 [==============================] - 208s 17s/step - loss: 0.1787 - acc: 0.7943 - val_loss: 0.1679 - val_acc: 0.8005\n",
      "Epoch 15/50\n",
      "13/13 [==============================] - 216s 17s/step - loss: 0.1718 - acc: 0.8123 - val_loss: 0.1598 - val_acc: 0.8365\n",
      "Epoch 16/50\n",
      "13/13 [==============================] - 201s 16s/step - loss: 0.1619 - acc: 0.8612 - val_loss: 0.1531 - val_acc: 0.8416\n",
      "Epoch 17/50\n",
      "13/13 [==============================] - 173s 14s/step - loss: 0.1609 - acc: 0.8378 - val_loss: 0.1470 - val_acc: 0.8502\n",
      "Epoch 18/50\n",
      "13/13 [==============================] - 157s 12s/step - loss: 0.1496 - acc: 0.8471 - val_loss: 0.1412 - val_acc: 0.8690\n",
      "Epoch 19/50\n",
      "13/13 [==============================] - 155s 12s/step - loss: 0.1471 - acc: 0.8600 - val_loss: 0.1379 - val_acc: 0.8604\n",
      "Epoch 20/50\n",
      "13/13 [==============================] - 154s 12s/step - loss: 0.1366 - acc: 0.8801 - val_loss: 0.1318 - val_acc: 0.8767\n",
      "Epoch 21/50\n",
      "13/13 [==============================] - 155s 12s/step - loss: 0.1362 - acc: 0.8708 - val_loss: 0.1285 - val_acc: 0.8664\n",
      "Epoch 22/50\n",
      "13/13 [==============================] - 156s 12s/step - loss: 0.1361 - acc: 0.8546 - val_loss: 0.1259 - val_acc: 0.8682\n",
      "Epoch 23/50\n",
      "13/13 [==============================] - 154s 12s/step - loss: 0.1197 - acc: 0.9104 - val_loss: 0.1231 - val_acc: 0.8733\n",
      "Epoch 24/50\n",
      "13/13 [==============================] - 146s 11s/step - loss: 0.1240 - acc: 0.8834 - val_loss: 0.1175 - val_acc: 0.8844\n",
      "Epoch 25/50\n",
      "13/13 [==============================] - 131s 10s/step - loss: 0.1145 - acc: 0.9165 - val_loss: 0.1165 - val_acc: 0.8853\n",
      "Epoch 26/50\n",
      "13/13 [==============================] - 131s 10s/step - loss: 0.1216 - acc: 0.8844 - val_loss: 0.1155 - val_acc: 0.8784\n",
      "Epoch 27/50\n",
      "13/13 [==============================] - 132s 11s/step - loss: 0.1084 - acc: 0.9093 - val_loss: 0.1111 - val_acc: 0.8878\n",
      "Epoch 28/50\n",
      "13/13 [==============================] - 127s 10s/step - loss: 0.1039 - acc: 0.9156 - val_loss: 0.1095 - val_acc: 0.8853\n",
      "Epoch 29/50\n",
      "13/13 [==============================] - 128s 10s/step - loss: 0.1066 - acc: 0.9175 - val_loss: 0.1095 - val_acc: 0.8818\n",
      "Epoch 30/50\n",
      "13/13 [==============================] - 194s 16s/step - loss: 0.0987 - acc: 0.9199 - val_loss: 0.1089 - val_acc: 0.8784\n",
      "Epoch 31/50\n",
      "13/13 [==============================] - 194s 16s/step - loss: 0.0995 - acc: 0.9164 - val_loss: 0.1047 - val_acc: 0.8827\n",
      "Epoch 32/50\n",
      "13/13 [==============================] - 206s 16s/step - loss: 0.0938 - acc: 0.9322 - val_loss: 0.1030 - val_acc: 0.8818\n",
      "Epoch 33/50\n",
      "13/13 [==============================] - 199s 16s/step - loss: 0.0907 - acc: 0.9205 - val_loss: 0.1014 - val_acc: 0.8853\n",
      "Epoch 34/50\n",
      "13/13 [==============================] - 213s 17s/step - loss: 0.0918 - acc: 0.9208 - val_loss: 0.0990 - val_acc: 0.8887\n",
      "Epoch 35/50\n",
      "13/13 [==============================] - 264s 21s/step - loss: 0.0887 - acc: 0.9342 - val_loss: 0.0978 - val_acc: 0.8878\n",
      "Epoch 36/50\n",
      "13/13 [==============================] - 378s 30s/step - loss: 0.0875 - acc: 0.9170 - val_loss: 0.0956 - val_acc: 0.8955\n",
      "Epoch 37/50\n",
      "13/13 [==============================] - 247s 19s/step - loss: 0.0856 - acc: 0.9363 - val_loss: 0.0969 - val_acc: 0.8896\n",
      "Epoch 38/50\n",
      "13/13 [==============================] - 224s 17s/step - loss: 0.0777 - acc: 0.9312 - val_loss: 0.0938 - val_acc: 0.8921\n",
      "Epoch 39/50\n",
      "13/13 [==============================] - 201s 16s/step - loss: 0.0837 - acc: 0.9205 - val_loss: 0.0930 - val_acc: 0.8938\n",
      "Epoch 40/50\n",
      "13/13 [==============================] - 201s 16s/step - loss: 0.0844 - acc: 0.9180 - val_loss: 0.0917 - val_acc: 0.8938\n",
      "Epoch 41/50\n",
      "13/13 [==============================] - 197s 16s/step - loss: 0.0731 - acc: 0.9353 - val_loss: 0.0917 - val_acc: 0.8938\n",
      "Epoch 42/50\n",
      "13/13 [==============================] - 210s 17s/step - loss: 0.0732 - acc: 0.9220 - val_loss: 0.0908 - val_acc: 0.8861\n",
      "Epoch 43/50\n",
      "13/13 [==============================] - 236s 19s/step - loss: 0.0718 - acc: 0.9440 - val_loss: 0.0923 - val_acc: 0.8896\n",
      "Epoch 44/50\n",
      "13/13 [==============================] - 186s 15s/step - loss: 0.0711 - acc: 0.9581 - val_loss: 0.0912 - val_acc: 0.8861\n",
      "Epoch 45/50\n",
      "13/13 [==============================] - 169s 13s/step - loss: 0.0704 - acc: 0.9449 - val_loss: 0.0893 - val_acc: 0.8887\n",
      "Epoch 46/50\n",
      "13/13 [==============================] - 183s 15s/step - loss: 0.0768 - acc: 0.9366 - val_loss: 0.0897 - val_acc: 0.8887\n",
      "Epoch 47/50\n",
      "13/13 [==============================] - 196s 16s/step - loss: 0.0723 - acc: 0.9305 - val_loss: 0.0861 - val_acc: 0.8990\n",
      "Epoch 48/50\n",
      "13/13 [==============================] - 154s 12s/step - loss: 0.0733 - acc: 0.9289 - val_loss: 0.0873 - val_acc: 0.8964\n",
      "Epoch 49/50\n",
      "13/13 [==============================] - 228s 18s/step - loss: 0.0691 - acc: 0.9568 - val_loss: 0.0878 - val_acc: 0.8998\n",
      "Epoch 50/50\n",
      "13/13 [==============================] - 211s 17s/step - loss: 0.0625 - acc: 0.9409 - val_loss: 0.0864 - val_acc: 0.8896\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    train_gen, epochs=50, validation_data=val_gen, verbose=1, shuffle=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAI4CAYAAACV/7uiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABuwklEQVR4nO3dd3hUVf7H8feZSe8JSYCQhCSAdKSDXdeu2F3Fspa17drXLeoW13XXXXf9ra59197Frth7V5AgSEdaIAkthSSkZzLn98edhCSEEglTks/reeaZmXtvZr4ZJZ855557jrHWIiIiIqHFFegCREREpOsU4CIiIiFIAS4iIhKCFOAiIiIhSAEuIiISgsICXcCeSk1NtTk5OYEuQ0REZK+YO3duqbU2reP2kA/wnJwc8vPzA12GiIjIXmGMWdvZdnWhi4iIhCAFuIiISAhSgIuIiIQgBbiIiEgIUoCLiIiEIAW4iIhICAr5y8h2xuv1UlpaSkVFBc3NzYEuR/YCt9tNUlISqampuFz6PioivUePDvCioiKMMeTk5BAeHo4xJtAlSTey1tLU1MSmTZsoKioiOzs70CWJiPhNj26y1NTUMGDAACIiIhTePZAxhoiICAYMGEBNTU2gyxER8aseHeCAulV7Af03FpG9rdlr+d9nqzjpvq94e+EGrLWBLqlnd6GLiIjsqXVltfz6xfnMKdhCalwElz/zHRMHJvPHaSMYm5UUsLrUdBEREemEtZYZ367j2Ls+Z9mGrdxxxr7M/v0R3HbqaArKajn5vq+4dsY81lfUBaQ+BXgv8fjjjxMWpg4XEZHdsbmqnoueyOeGVxayb1YS7/7qYE4dn4nbZZg+OZtPf3soVxw2iLcXbeSw//uUf7+/nJoGj19rVIAHsSOOOIILLrigW17rzDPPpLi4uFteS0SkJ3trwQaO+s/nfLWylD+fMIKnL5rCgKTodsfERYbx26OH8fGvD+GYUf245+OVHPp/n/L8nHU0e/1zflxNshDX2NhIRETELo+Ljo4mOjp6l8eJiPhTZV0Tc9aUM2t1GQVlNYzJTGJqXh/2zUokMszdbe/T7LWsKa1mS23TDo+xFp6dvZbX5q9nTGYid5wxlsHpcTt93czkGO6aPo4L9s/hb28t5fqXFxIbGca0MRndVvuOKMCD1AUXXMBHH30EwBNPPAHAY489xoUXXsjTTz/NM888w2effcaVV17JbbfdxqWXXsrHH3/M+vXr6d+/P9OnT+fPf/4zkZGRgNOFfvHFF+PxeNo9/+yzz7jqqqtYtmwZw4cP57///S+TJk0KzC8tIj1eZV0T+QVOYM9aXc7i9ZV4LUSEuchMjuajZZuxFiLDXEwYmMzUvD5dDvSaBg/LNm5lyYYqlqyvYsmGKpZvrKK+ybvLn3W7DL86Yh8uP2wQ4e7d76Qel53MS7/Yj4+XbebQoem7/XN7otcF+F/eWMyS9VV+f98RGQn8+YSRu338XXfdxerVq+nfvz933XUXAFVVTt3XX389//znP7nvvvsAZ6BFeno6zz77LH379mXBggVcdtllhIeH85e//GWH7+H1ernxxhu56667SEtL41e/+hVnnHEGK1as0PlyEek2FbWNPPdtIW8v3NAusMdnJ3H14UOYmteHsVlJRIW7qaxt4tvWgC/jzg9/aA30fbOSSIja8d+mZq9lbVkta8pqaLnKKzE6nJEZCZw7ZSAjMhJIi4/EsON5QTKTo8lJjf1Rv6cxhsOH9/1RP/tj6K90kEpMTCQiIoLo6Gj69esHQH19PQCXXXYZ55xzTrvjb7311tbHOTk5rFq1ivvvv3+nAW6t5T//+Q/jx48H4Oabb2bq1KmsWrWKoUOHdvevJCK9zMrN1Tz21Rpe/q6I+iYvEwcmbxfYHSXGhHPkiL4cOcIJworaRr5dU86s1eXMK9zChsr6nb7nkL5xnDR2ACMzEhiRkUD/xKgeO5FXrwvwrrSCg9XkyZO32/bQQw/x8MMPU1BQQE1NDR6PB693591Fxhj23Xff1ucZGc45m02bNinARaSdrfXOueP4qPCdHmet5fMVpTz65Ro++6GEiDAXp4wdwIUH5jCsX0KX3zcpJoKjRvbjqJH9flTdPVmvC/CeIDa2fffOiy++yBVXXMFtt93GIYccQkJCAi+++CJ/+MMfdvo6LpcLt3vbN+CWb6m7Cn4RCR7NXkt1vQevtSTFdP+aDxsr6/nvZ6t49tt1NHq8JEaHk5kc7bvFtN4PSIpmfmEFj361hpWbq0mLj+S6I/fhnCnZ9ImL7NaaxKEAD2IRERG7tYra559/zrhx47juuutatxUUFOzFykR6rqZmLxsq6inaUkvRljqKttRSUddEXGQYCdHhJESFkxAd5rsPJyEqjJiIMGoaPVTVNVFV33LfRFWdh6r6JmoaPGQmRzOifyIjMhJIid31lSPWWgrL61iyoZIl66soqqhrfb2quia2+t5na5trj6PD3Z2Ga2ZyNNkpMSTvxvu22FRVzwOfOsHt9VpOHT+AvLS41s9lVUkNn/1Qst3AsJEZCdxxxr4cP6Z/t44il+0pwINYbm4un3zyCatWrSIxMZGmps4vfxg6dCiPPPIIr7/+OqNGjeLNN9/klVde8XO1IsGlvqmZ+YUVzF5dzpbaxp0eW1XfRFG5E9Ybq+ppexmvyzjdxjUNHjw/4vreMJchOsLN1vptQds/MYoR/Z1ztCP6JzC8fwLVDZ7WEdNL1lexdENVazi7XYZ+CVGtXxiyUmK2+yIBsL6irjVgv1tXQWVd+78Zg9JiW0d1T8lLIT0+art6Owb3aeMzueKwwWT3idnuWGstZTWNrV90+iVEMWFgco895xxsFOBB7Ne//jULFy5k3333paamhscee6zT4y677DIWLlzIhRdeiMfjYdq0adx8881cddVVfq5YJHBaArtl9PJ36ypo9HgxBhJ2cd42NsJNZkoMUwf1adNyjSYrOYZ+iVGEu11Ya6lram7XCm5pZdc0epwW+nat83Ciwl0YYyivaWRpm8ualqyv4tMfSrab9CMmws3w/gmcMn5Aa8jv0ze+0wFfu1JV30TxljqKttSxcnM1364p4/X563lm9joABqfHMTUvhal5fRiSHs9z367breBuYYwhNS6S1LjIgM4J3luZYFhRZU9MnDjR5ufnd7pv6dKlDB8+3M8VSSDov3Xv4Wn2sqGyvrXVt7aslvy15e0Ce2RGAlNznZbmpNwUEqN3HuCBUt/UzIpN1SzdUEVsZBgjMhIYmBKDy7X3WrCeZi+L1le1ftGZs6acmkbnVF2Yy+xWcIt/GWPmWmsndtyuFriI7BXWWr5cWcqCokoSonZ0/jicyDAX1S3nj9u1bp1tFbWNFLc5J72xqr5dq9VlnHkWzps6MOgDu6OocDejMxMZnZnot/cMc7sYm5XE2KwkfnHIoNZAX1RcycFD0hTcIUQBLtJDVdQ28unyEqaN6U9YF2aU2lPWWr5YUcp/PvyB79ZV7PHrGQP9E6LITI5hSm7KdgO0+iVGERGmZR1+rLaBLqHFrwFujDkGuAtwAw9ba2/rsH8g8CiQBpQD51pri/xZo0hP4PVarnpuHl+sKOX1+cXce/Z4YiP37j/3jsGdkRjF304exUljM6hv8m7Xsm45f1zf1Ex8Jy30RN/zuKgw3HuxS1kkVPktwI0xbuA+4EigCJhjjJlprV3S5rD/A5601j5hjPkJ8A/gZ/6qUaSneOiL1XyxopTjx/Tn3UUbOfPBb3j0gkmdjjreUzsK7p9OzGy9jCg+CtLidS2wSHfyZwt8MrDSWrsawBgzAzgJaBvgI4CWi5k/AV7zY30iPcL3hRXc/t5yjh3Vj3vPGseny0u44tnvOOW+r3ni55MYnB6/R6/v9VpKqhso2lJLQWktz8xeu8PgFgkq9VXQWA2x6eDuQvzVV0L5GihfDU11MOw4iE7ee3XuJn8G+ACgsM3zImBKh2O+B07F6WY/BYg3xvSx1pa1PcgYcylwKUB2dvZeK1gk1Gytb+LqGfNIj4/ktlPHYIzhsGHpPH/pflz4+BxOvf9rHjpvIlPy+uzyteqbmvnshxJWbq5uM6lJHcVb6mhs3jZ5h4Jbgl7xdzDnEVj0EnjqwbicEE/oD/G+W8tjV9i2sN7iu68ta/96b0XDmJ/CpIuh/76dv6cfBNsgtt8A9xpjLgA+B4qB7aYis9Y+CDwIzmVk/ixQJJjd9PpiCstref6y/UiM2TYSe3RmIq9evj8XPPYtP3vkW27/6RhOGjtgu5+31rKgqJIX5xYyc/56qnyTj/SJjfDNJJbAUSP6tg4gy0qJZmCf2C4tuyg+DdWw6iNY9hb88B5ExMLA/X23AyF1iDOCryushaZaqNuy/a1hK+zpZcN9R0LOQRC2+zO6BUxTPSx+FeY8BMVzITwW9j0L+o2GrRth63rnfstaWDcL6sq3/axxQUImpOTC8BMgJQ+Sc5375kaY+xgseBG+exIyJ8PkS2DESRDm39NE/gzwYiCrzfNM37ZW1tr1OC1wjDFxwGnW2gp/FSgSyl75rohX5xXzqyP2YVJOynb7s1JieOWXB3DJU/lcM2M+6yvq+cUheRhj2FxVz6vzinlpbhErNlcTGebimFH9OG18JhNzkomJCLbv+iGqugR+eMcJ7VWfQHMDRKfA0GOdluGaz2Hhi86xMam+MD8Acg6AuH6wdYNzq1rfPoSqNkDNZieom3c+69wei0yAIUfBsONh8BEQtZsLlFgL1guuvdxLs6UA8h+F755yQjl1Hzj2X7DvdIjayeV6TfXOZ+tthqSsnYfxgPFw5F9h/rMw52F45RJ490YYfx5M/Lnz837gt4lcjDFhwA/A4TjBPQc421q7uM0xqUC5tdZrjLkVaLbW3rSz19VELgL6b72mtIZpd3/ByAGJPHfJ1J2O2m7wNPObFxfwxvfrOWlsBlV1TXz2QwleCxMGJnP6hEyOH9N/l7OX7ZK3GVZ/CmWrIL4vxGdAfD/n5g6N67Q7VbdlWxdr+Rqnm7Wq2Ol6DYuC8Ojt78EJ53WzAAtJ2TBsmhOCWVO3nY+11nndtV9BwVew9muoXLeDQgzEpfs+0wzncUyKc242Ksm5b3uLjHdalj+W1+PUv+xNWP4O1JaCOwJyD3F+j6HHQWya86WiYxd0+WooLwBPHSRmOS3ZlNz2LdvkHAiPAk8jVPu+lHT8wlK9yfkSsCNNdU5r27ic89STLoHcg7vek9Glz8ULaz51uuiXv+1sO/Np5zPpJjuayMWvM7EZY44D/oNzGdmj1tpbjTG3APnW2pnGmNNxRp5bnC70K6y1DTt7TQX4jj3++ONcfPHFeDxON+inn37KYYcdRmFhIZmZmTv8OWMMTz31FOeee+4evf8FF1xAUVERH3744R69zu4Itf/WpdUN3PLGEob1j+fiA/P26DrmRo+X0x74mnXltbxzzUFkJEXv8me8Xsu/31nIkq9mkhPTwNTsGCZkxJAa5XVaIp465765AdKGw+DDoc/g3ftDWL4a5j0D3z/nBNt2jPOHPr4fJGRAwgDnj3fLH/XkHKc7OVCsherNHQKoTSDVbWl/fHx/53ewzb7Pzndrqtv2GJyu25bQ7jtq90OlYh2s/QbqK3zv5fsiFNc3cF+EvM1QONvpSVj2ptPqxTiB3tzmT7YrHJIHbgvpiFjn2JbPtKGyzYsap4VcX7H9+7kjnN99l7+zgYH7wYQLIXH7U0R7XUUhfPcE7HclRCd128sGxUxs1tq3gbc7bLupzeOXgJf8WVNvsv/++7NhwwbS09O79XWffvppfvazn9Hxy+Bdd92lpUk7sXh9JZc+OZeNVfXM/H49r89bz99PHcWEgdt3e++O/3t/OQuLK/nvuRN2K7ypLceV/yi/XfogRGwCD7Dad2vhjnRaQ64wmPu4sy0x2wnywYc7rZq23ZGNNbDkdSe4137ptIAGHQ5H3wrZ+zmB2NLl265lVewEQcdQjOvXoYWWu+35rkb/ehqg9AfYvBQ2LYaSZc554bBo53dqex8W6bSQG2t8AV3gBEtTzbbXM65trcaRp2zfaozYxcxl1kJz048/b5yU7dyCicu97Xz9UX+DzUtg2dtO+KbkbfsylpC549He1vp6M9p8QarZ7Awua/ly1zLALCZl77aiu0tSFvzkj357O53Y6kUiIiLo16+f394vMdF/00OGijcXrOc3L35PckwEr11+AJuq6rnp9UWc9sA3nD0lm+uPHtZu8NmufPZDCQ9+vppzp2ZzzKhd/LctXwOz7od5TzuBNugncOK90GdQ+y7fsChwtekR2FIAKz9ybgtfcgbwGDdkTXZeo7IQFr0KjVudP9yH3+QMFkrI2PYa8f2g/5gd19bSLd3a4i1w7ld+5HSnthWdvC1AU3KdcNu60QmRTUugbKXTGganBZg6xDlvW1+5rXfB07DtcXOj08Jr6QXIOah9CCVm7dmgLWNCY9DXj2WMM7it78iu/1xMinPL3K5xKbvDWhvStwkTJtgdWbJkyQ73BbsHH3zQJiQk2Lq6unbbb7vtNpuVlWU9Ho+9+OKLbV5eno2KirK5ubn2xhtvtPX19a3HPvbYY9btdrc+/+STTyxgCwsLW7d9/PHHdvTo0TYyMtKOHj3afvzxxxawTz31VOsxv//97+2wYcNsdHS0zczMtJdddpmtqKho95ptb+eff7611trzzz/fHn744a2v4/V67e23325zc3NteHi4zcvLs3feeWe732/gwIH2T3/6k7366qttcnKyTU9Pt9dee61tamra6ecV7P+tm5u99l/vLrUDr3/Tnnb/V7a0aIW1r15u7fs32ZqNq+xf31hsc29400746wf29fnF1uv17vI115XV2Al/fd8eecentq7Rs5MDv7X2+Z9Ze3OStX/pY+0rv7B2w8If94t4Gq1d86W1H/7F2v8ebO2fE6z9W3/ndyn4ytrdqLvLGqqt3bjI2iVvWPvlXda+ca21T5xo7Z2jnd/pzwnO7T9jrH12urUf3mLtghet3bTE2qaGXb9+s8fa5ubur1ukm+CcZt4u/9QCD1JnnHEGV199Na+//jpnnnlm6/Ynn3ySc889F2MM6enpPPvss/Tt25cFCxZw2WWXER4ezl/+8pfdeo/169czbdo0zjjjDGbMmEFxcTHXXHPNdsdFR0fz4IMPkpWVxapVq7jiiiu4+uqreeKJJ9h///259957ufLKK9mwYUPr8Z25//77+dOf/sRdd93FYYcdxkcffcS1115LfHw8F110Uetx99xzD9dffz2zZ89m3rx5nHPOOYwaNardMaGkqr6JX82Yz0fLNnPWxAHckjWH8Cd+6pxHbG4g5qu7+OM+R3PuSWdzzZx4rn5uHi/PLeJvJ48iKyUGr9dSUFbTugRly/3mrQ1Ehrl45uKp7ZearK+E9fOca19/eNfpoo5KhAOugcmXOde7/ljucGdEdM4BTku7ttzpht6b56wjYnfcwvM0Ot3wsWkQGffjXn9vj4oW2Ut6X4C/cwNsXOj/9+03Go69bdfH+SQmJnLSSSfx5JNPtgZ4fn4+S5Ys4ZVXXsHlcnHrrbe2Hp+Tk8OqVau4//77dzvA77//flJTU3nooYcICwtjxIgR/P3vf+eEE05od9wf/7jtnE5OTg7/+Mc/mD59Oo899hgRERGtXeW76p6/7bbbuOqqq7j00ksBGDJkCMuXL+fWW29tF84HHXQQN9xwQ+sxjz32GB9++GFwBbjX65zPTRiw03Nza0pruPiJOawtq+XOIxM5ufAPmHe+hLzD4MS7nfOrcx+HuU+Q88O7vJacS/7oU7hm+UiOvLOMYf0S+GHTVmrbLPc4pG88Bw1JY0RGAofkxjG4aRnMmgvrv3NCu2zFtgJaLqEZe86PD7idiflx5+27TViE080t0gv1vgAPIeeffz4nnngimzdvJj09nSeffJLJkyczdOhQAB566CEefvhhCgoKqKmpwePxdGnQ2JIlS5g8eTJhYdv+NzjwwAO3O+6VV17hP//5DytXrqSqqgqv10tjYyMbN24kIyNju+M7U1VVRVFREQcffHC77Ycccgh33XUXtbW1xMQ4g4HGjh3b7piMjAzWrFmz27/XXldbDq9cCis/gKSBMOo0GH36di3Ez34o4apnvyPMWD7Yfwm5s/7tDAo78R4Y97Ntwf+TP8LBv4OlMzFzHmbSijv4KjyKb+MOY37jEK7IbCIzupG+4bUkUo27vgLKKqBoC3y80bm8B5wRugMmwL5nQsZ4yBgX+IAVkb2m9wV4F1rBgXbUUUeRmprKs88+yxVXXMGMGTO4+eabAXjxxRe54ooruO222zjkkENISEjgxRdf5A9/+EO31jB79mx++tOfcuONN3L77beTnJzMrFmzOP/882ls3DsTRkREtB/wY4wJntHsRfnw4gXO9agHXOv05nx1F3x5h3O51ajTKM09gX/ObuCl74r4SWoV98c9QmT+HGfyi2n/6fzylrAI50vA6NNh40LMnIeZsuAFpjS9A5U4g6xarueNSoLETKdXJ6G/E9QZ451BY6EwUldEukXvC/AQ4na7Oeecc3jqqafIy8ujsrKS6dOnA/D5558zbtw4rrvuutbjCwoKuvT6I0aM4KmnnqK5uRm32zkP+NVXX7U75ssvvyQ1NZW//e1vrdteeqn9lX4tgdv2dTpKSEggMzOTzz//nGnTprVu/+yzz8jNzW1tfQcta+Hbh+C93zuh+fP3nNmYAGpKYclrNC94CfcnfyP1k7/xM+8gzswax4TyNzFNUXDK/2DMmbsXsP1Gwwl3wVG3OpflRCdDeIzCWUTa0QTGQe68887ju+++489//jPTpk0jJcXpEh06dCgLFy7k9ddfZ9WqVdx111288sorXXrtX/7yl5SUlHDppZeydOlSPvroo+1a8EOHDqWkpIRHHnmE1atX8+STT3L//fe3OyY31zkHOXPmTEpKSqiuru70/W688UbuueceHnroIVasWMH//vc/HnjgAX7/+993qW6/a9gKL10I7/zWmTryss+3hTfgje7Dy65jOGDTb9mv/h5eSf0Fw/rGMHHzS5jBR8AVs51pHLsawJFxTks7IlbhLSLbUQs8yI0ZM4axY8cyf/781u5zgMsuu4yFCxdy4YUX4vF4mDZtGjfffDNXXXXVbr/2gAEDeOONN7j22msZO3YsQ4YM4e677+bwww9vPWbatGn84Q9/4Pe//z3V1dUccsgh3H777Zx99tmtx0yaNIlrrrmGyy67jJKSEs4//3wef/zx7d7vl7/8JTU1Nfz973/n8ssvJysri9tuuy24Bqd1tGkxvHCec43yEX+B/a9ud430rNVl/O2tJSwqrmLfzET+dPY0Juac5+xs2AoRcQpfEdkr/DqV6t6gqVQFuv7fetbqMgpKa/jJ8HTS46M6P2j+s/Dmdc5iDac/5lw6BTQ1e5m9upynZhXw3uJN9E+M4vpjhnHivhm4djIHuYjIjxEUU6mKBINFxZWc/+i3NHi8GAPjspI4amQ/jh7Zj9zoeljymjPj2LqvnVm5TnuEmog+fLZwA+8v3shHyzaztd5DbISb3xy1DxcdmEd0hK4lFhH/UoBLr1Je08hlT80lJTaC/5w5ltlryvli8WqWv/cQQz/8miz3IsJopj5xEE0H/5m3Y07mvZfX8eXK72j0eEmOCedoX9gfODhVwS0iAaMAl17D0+zlyme/o6S6gZcvHsvo2i+ZUvoSV1e9DxH1VEf1562w03hoyzgWbcqG9w2wlMzkaM6dMpCjRvZl4sBkwtwa+ykigacAl56vuQk2L+W9d99i2trZ3J+ynqQnfAtexKbB+PNg1OnEZU7iJJeLg2sa+WjZZjZvrefQfdIZ3j8eo4FoIhJkenyAW2v1x7eH63Qg5povnLWK138HG74HTz3HA7WR8cSkTYJ9T4CcA51z3B2WO0yOjeD0CTteL11EJBj06AAPDw+nrq4u+CcJkT1SV1dHeHibJTiL5sKTJzmzl2WMpXT4ufzj+2ga+47jjstOhjCdtxaR0NejAzw9PZ3i4mIGDBhAdHS0WuI9jLWWuro6iouL6du3r7OxqQ5evQzi+8Mvv6LcG8NJ93yJN9oy87wDCVd4i0gP0aMDPCEhAXCWzWxqagpwNbI3hIeH07dv39b/1nz4F2c1rvNexxORwFWPfUtJdQMvXrYfafGRgS1WRKQb9egAByfEW/+4S8+2+jOY/YCz5nXeofzzrSV8tbKM208fw75ZSYGuTkSkW+l6GOkZ6ivhtcuhz2A44mZen1/MQ1+s4fz9BvLTiVmBrk5EpNv1+Ba49BLv3ghb11Pzs3e4/d01PPlNAZNzUvjjtBGBrkxEZK9QgEvoW/YWzH+GVcN+wTkzati0tYzzpg7kN0cPJVyTrohID6UAl9BWXULz61dRHDGYY+bvz+D+Efz3ZxMYq3PeItLDKcAlZHk8zRQ9cSkZtZVc6b2B3x03mgsPyNFUpyLSK/j1L50x5hhjzHJjzEpjzA2d7M82xnxijJlnjFlgjDnOn/VJ6FhYVMnd//krOSUf82ryhdz3q3O55OA8hbeI9Bp+a4EbY9zAfcCRQBEwxxgz01q7pM1hfwResNY+YIwZAbwN5PirRgkNC4squfz+mbwT8T/K+0zgjMtvw7jVmSQivYs/myuTgZXW2tXW2kZgBnBSh2Ms0HLRdiKw3o/1SQhoavZy00vfckfk/4gNh5RzHlZ4i0iv5M+/fAOAwjbPi4ApHY65GXjfGHMVEAsc0dkLGWMuBS4FyM7O7vZCJUhZyyevPMQ9W/5FpimFY++BlLxAVyUiEhDBdsLwLOBxa20mcBzwlDFmuxqttQ9aaydaayempaX5vUgJgM3LqHtkGkct/h02MgEueNtZBlREpJfyZ4AXA22nxMr0bWvrIuAFAGvtN0AUkOqX6iQ41VfCuzdiH9if5uL53Gp/TsTlX0DOAYGuTEQkoPwZ4HOAIcaYXGNMBDAdmNnhmHXA4QDGmOE4AV7ixxolWHi9MO9puGcCzHqA1VmnclDd/5F73K/omxQX6OpERALOb+fArbUeY8yVwHuAG3jUWrvYGHMLkG+tnQn8GnjIGPMrnAFtF1hrrb9qlCBRtQFe+BkUzYHMSZSf8gynPlPJ0Jx4pk/SvOYiIuDniVystW/jXBrWdttNbR4vAdQ32pt5vfDqpbBpMZz8AIyZzp9mzKeusZl/nDYal0truouIQPANYpPebtb9sOZzOOY2GHs2Hy4r4a0FG7j68MEMSlPXuYhICwW4BI+Ni+Cjv8DQ42H8eWytb+JPry9iaN94Lj14UKCrExEJKpoBQ4JDUz28cilEJcGJd4Mx3P7ecjZW1XP/OeOJCNN3TRGRthTgEhw+/itsXgznvASxqeQXlPPUrLVcsH8O47KTA12diEjQUYBL4K3+FL65FzvpEsr6H0xRYQU3vLKQjMRofnPU0EBXJyISlBTg4nf1Tc18tHQzhVtqKSvZyC+XXsRWVyYnzTqQii8+BMBl4JELJhEbqf9FRUQ6o7+O4ne/fvF73lqwAbA8GHUvCWzhgcz7OL3fPmQmR5OZHMOQvnEM7BMb6FJFRIKWAlz86rMfnMvCLj90EFelziX6zW/g8D/zh4POCnRpIiIhRUN7xW/qm5q56fVF5KXGcs3ESKI/uAGy94cDrgl0aSIiIUctcPGb+z9dxdqyWp75+UQiZ17gbDzlv+ByB7QuEZFQpAAXv1hdUs1/P13FSfv254CCe2Dd13DKg5A8MNCliYiEJHWhy15nreVPry8iMtzw98RX4et7YOJFMOaMQJcmIhKy1AKXvW7m9+v5amUpb4z4hNhvH4YJF8Bx/wdGC5OIiPxYCnDZqyrrmvjrG0v4Z/IbjF49A8afD8ffCS51/oiI7AkFuOxV/35/OT9reJYzm1+B8efBtP8ovEVEuoH+kspes6CoguQ5d3BN2Csw7lyYdpfCW0Skm+ivqewVzV7Lwmd+z6/CXqZx9Nlwwj0KbxGRbqS/qLJXLHj2D5xT9wzrsk8h4pT7FN4iIt1Mf1Wle1lL9ft/Z9zK+/gi5giyzn9Y4S0ishfoL6t0H2ux7/+JuK//yaveg8m64DGMW+MkRUT2BgW4dA9vM7xxNeabe3jccxQbDvs3OekJga5KRKTHUvNI9pynEV69FBa/yv84lQ8yLub5Q4YEuioRkR5NAS57prEWXjgPVn7A0wkXc3fFkbxz5jjcLs2yJiKyNynA5cerr4Rnp8O6b/hi2J/44/zh/Ou0kWT3iQl0ZSIiPZ5fz4EbY44xxiw3xqw0xtzQyf47jTHzfbcfjDEV/qxPuqCmFJ44AYq+Zf0R93HRopEcOaIvP52YGejKRER6Bb+1wI0xbuA+4EigCJhjjJlprV3Scoy19ldtjr8KGOev+qQLKovhqZOhYh1NZzzDRe/FkhBVzz9OHY3RAiUiIn7hzxb4ZGCltXa1tbYRmAGctJPjzwKe80tlsvu2FMBjx0DVBjj3Fe4oyGHphipuO3UMqXGRga5ORKTX8GeADwAK2zwv8m3bjjFmIJALfOyHumR3la2Cx46D+io4fyZzGM5/P1vF9ElZHDGib6CrExHpVYL1OvDpwEvW2ubOdhpjLjXG5Btj8ktKSvxcWi+1eRk8dix4GuCCN9naZzS/en4+Wckx/HHaiEBXJyLS6/gzwIuBrDbPM33bOjOdnXSfW2sftNZOtNZOTEtL68YSpVMbF8HjxzuPL3gL+o3mr28uYX1FHXecsS9xkbqYQUTE3/wZ4HOAIcaYXGNMBE5Iz+x4kDFmGJAMfOPH2mRH1s+HJ6aBOwIueBvSh/He4o28kF/ELw8dxMSclEBXKCLSK/ktwK21HuBK4D1gKfCCtXaxMeYWY8yJbQ6dDsyw1lp/1SY7UJQPT5wIEfFw4duQOpgPl2zi1y98z8iMBK45fJ9AVygi0mv5te/TWvs28HaHbTd1eH6zP2uSHVj7DTzzU4hNhfNnYhOzuO/jFfz7gx8YPSCRB382kYiwYB1CISLS8+nkpWxv9Wfw3HRIGADnz6Q2Kp3fPjuPtxZu4OSxGdx22hiiwt2BrlJEpFdTgEt7Zavg2TMheSCcN5PCpnguuf9rfti0ld8fN4xLDsrTZC0iIkFAAS7beL0w82pnwNrPXmNWSRiXP/MVTc1eHr1gEocOTQ90hSIi4qMAl23mPQVrv8SecDdPL27gL2/MY2CfGB46byJ5aXGBrk5ERNpQgItj60Z4/094Bx7IHwrG8dycxfxkWDr/mT6WhKjwQFcnIiIdKMDF8fZvsZ56/uS9hOfmFHL5oYP49VFDta63iEiQUoALLH0Dls5kRsLPeWZFOLecNJLz9ssJdFUiIrITCvDerq4C75u/Zq07j5tLD+OOM/bl1PFa01tEJNgpwHu5unf+RERNCb/2XM3d50zm6JH9Al2SiIjsBgV4L7Z50UekL3iSx+w0rjt/OgcOSQ10SSIispsU4L3UqvWlhL18FcWkMfZn/2LcYIW3iEgo0WTWvdCi4ko+fuh3DLTFNB93J+MGDwh0SSIi0kVqgfcyi4oruenBF3jBvMbWoT8le/IJgS5JRER+BAV4b+BphPoKSjZv5K5nP+evrscxUUnEn/SvQFcmIiI/kgK8pyn8Fj65FWrKoG6Lc2uqASANeKjluOMegZiUQFUpIiJ7aLcD3BhzAVBrrX2hw/YzgChr7ZPdXJt0VU0pPP8z5/GA8dBvNEQn441K4vnF1XyzvpkLDh/H+NGjIW1oYGsVEZE90pUW+PXAVZ1sLwXuBRTggWQtvH4l1JXDJR874e1zx3vLubdwJX84bjjjD84LYJEiItJduhLgOcDKTrav9u2TQJrzMPzwDhxzW7vwfuW7Iu79ZCXTJ2Vx8UG5ASxQRES6U1cuI6sEOkuAQUB195QjP8qmJfDeH2DwkTDlF62b8wvKueHlhUzNS+GWk0ZhjBYmERHpKboS4O8Atxtj+rdsMMZkAP8E3u7uwmQ3NdXBSz+HqEQ4+QHwhXRheS2XPTWXjKQo/nvuBCLCdMm/iEhP0pW/6r8DYoFVxph8Y0w+Tpd6rG+fBML7f4KSpXDKAxCXBsDW+iYuemIOTc1eHrlgEkkxEQEuUkREuttunwO31pYYY8YB5wDjfZvvB56z1tbtjeJkF5a/A3Megv2uhMFHAOBp9nLVc/NYVVLDkz+fzKC0uAAXKSIie0OXrgO31tYDj/huEkhVG+C1y50Ba4ff1Lr5iW/W8unyEm49ZRQHaH5zEZEea7e70I0xNxhjLupk+0XGGHWh+5PXC69eBp56OO1RCIsEwFrLM7PXMj47iXOmDAxwkSIisjd15Rz4pcDyTrYvBS7rnnJkt3xzD6z5zLlkLG2f1s35a7ewuqSG6ZOzA1iciIj4Q1cCPAMo6mT7emC3lrMyxhxjjFlujFlpjLlhB8ecYYxZYoxZbIx5tgv19Q7F38FHt8DwE2H8ee12zfi2kLjIMI4f3X8HPywiIj1FV86BbwZGAwUdto8Bynb1w8YYN3AfcCTOF4E5xpiZ1tolbY4ZAtwIHGCt3WKMSe9Cfb3DF/+G6BQ44a7WS8YAquqbeGvhek4Zl0lspKa4FxHp6brSAn8FuNM3Eh0AY8x44N/AS7vx85OBldba1dbaRmAGcFKHYy4B7rPWbgGw1m7uQn09n7VQONsZcd5hIZKZ89dT3+Rl+qSsABUnIiL+1JUA/wNOy3muMabUGFMK5ON0of9+N35+AFDY5nkR23e97wPsY4z5yhgzyxhzTGcvZIy5tOVa9JKSki78CiFuSwHUlEDWpO12PT+nkGH94hmTmej/ukRExO+6ch14DXCoMeYnwATf5rnW2o+7uZ4hwKFAJvC5MWa0tbaiQy0PAg8CTJw40Xbj+we3ojnOfWb7AF+8vpKFxZX8+YQRmi5VRKSX6NLJUmNMMtAXcAMRwIHGmAMBrLW37OLHi4G2/buZvm1tFQGzrbVNwBpjzA84gT6nK3X2WIXfQkQcpI9ot/mFOYVEhLk4ZdxujSUUEZEeoCvrgU8C3gUMkACUAOlALbAB2FWAzwGGGGNycYJ7OnB2h2NeA84CHjPGpOJ0qa/e3Rp7vKI5zjrfLnfrpvqmZl6dV8wxI/tpylQRkV6kK+fAbwdeBlKBOuAAYCAwD2et8J2y1nqAK4H3cK4df8Fau9gYc4sx5kTfYe8BZcaYJcAnwG+ttbsc4d4rNNbCpkXbdZ+/u2gjVfUeDV4TEellutKFPhb4pbXWa4zxAhHW2tXGmOuBR4FXd/UC1tq36bBymbX2pjaPLXCd7yZtrZ8HXg9kTm63ecacdWSnxDA1r0+AChMRkUDoSgu8GWjyPd7MtvPZpTgtcdmbir517tu0wAtKa5i1upwzJ2XhcmnwmohIb9KVFvgCnFb4SmAW8HtjjAvn2u3OpliV7lQ4B1IGQey2lvYL+YW4DJw2PjOAhYmISCB0pQV+K+DxPf4TzgC2d4CDgKu7uS5py1pnAFub1ren2cuLc4s4bGg6/RKjAliciIgEQleuA/+wzeMCYKQxJgXY4jt3LXtLxVqo2dxuApdPlpdQsrWBMzV4TUSkV9qjSbOtteXdVYjsRGHLBC7bBrA9P2cdafGRHDZM08WLiPRGXelCl0Ap+hbCY1sncNlUVc/HyzZz+oRMwt36Tygi0hvpr38oaJnAxe10mLw0twivhTMmqvtcRKS3UoAHu6Y62LiwdQCb12t5Ib+QKbkp5KbGBrg4EREJFAV4sGuZwCXLOf89a00Za8tqmT5ZrW8Rkd5MAR7sCn0TuAyYCMCTX68lPiqMY0f1D2BRIiISaArwYFc0B5JzIS6NBUUVvLt4Ixfun0NUuHvXPysiIj2WAjyYtUzg4us+/9e7y0mOCeeSg/MCXJiIiASaAjyYVayD6k2QOYkvV5Ty5cpSrjhsMPFR4YGuTEREAkwBHsyKnAlcbOZE/vXeMgYkRXPuVK0bIyIiCvDgVjQHwmN4tySVBUWVXHvEEJ37FhERQAEe3Aq/xWaM4/YPVjEkPY5TteqYiIj4KMCDVVMdbFzAUvdQVpfW8Nujh+LWmt8iIuKjAA9W6+eD18Nj69IYn53EkSP6BroiEREJIgrwYOUbwPZxdQ7XHzMMY9T6FhGRbfZoOVHZe5rWzmYjfRk9dDBT8voEuhwREQkyaoEHI2tpWDOLuc2D+N3RwwJdjYiIBCEFeBAqKV5JXFMpnoyJjMhICHQ5IiIShBTgQeij998C4IBDjwtwJSIiEqwU4EFmTWkNdWu+odFE0n+fiYEuR0REgpRfA9wYc4wxZrkxZqUx5oZO9l9gjCkxxsz33S72Z33B4I4PfmCCayVkjAO35jwXEZHO+S3AjTFu4D7gWGAEcJYxZkQnhz5vrR3ruz3sr/qCwfqKOj5YUMBIVwEROVMCXY6IiAQxf7bAJwMrrbWrrbWNwAzgJD++f9CbMaeQkaYAt/VA5uRAlyMiIkHMnwE+AChs87zIt62j04wxC4wxLxljsjp7IWPMpcaYfGNMfklJyd6o1e+amr3M+HYdP+273tmQOSmwBYmISFALtkFsbwA51toxwAfAE50dZK190Fo70Vo7MS0tza8F7i0fLtnE5q0NHBa7FpKyIV5Tp4qIyI75M8CLgbYt6kzftlbW2jJrbYPv6cPABD/VFnBPz17LkERIL/kGBh4Q6HJERCTI+TPA5wBDjDG5xpgIYDows+0Bxpj+bZ6eCCz1Y30Bs7qkmq9WlvHnAd9iGqpg8iWBLklERIKc3+ZCt9Z6jDFXAu8BbuBRa+1iY8wtQL61diZwtTHmRMADlAMX+Ku+QHpm9jqiXc3st/l5yDkIBvSajgcREfmR/LqYibX2beDtDttuavP4RuBGf9YUaPVNzbw0t4jfZy7EvXkDnHxvoEsSEZEQEGyD2HqdN75fT1VdA6fVvwz9RsOgwwNdkoiIhAAtJxpgT89ex7lJS4ipWgVHPgJa91tERHaDWuABtKi4ku8Lt3Bl5JuQNBBGnBzokkREJEQowAPomdlrOSD8B/pWLoD9rwK3OkRERGT3KDECpKq+idfmreeVpPehuQ+MPSfQJYmISAhRCzxAXv2umCxPAcO3fgNTfgERMYEuSUREQoha4AFgreXpWWu5MeE98MbCpF63aqqIiOwhtcAD4Ns15dRsLuDQxs9hwvkQkxLokkREJMQowAPg6dnr+GXUu84VY1MvD3Q5IiISghTgflaytYFvFq3gTNcnmFGnQ1KnK6aKiIjslALcz17IL+Qs3iPCWwcHXBPockREJERpEFsXNHstSzdU0dTs3eExXgvVDR6q6pqoqm+iqs7ju2+iqt5D/g9FvBf5AQw6GvqO8GP1IiLSkyjAu+DluUX87uUFXf65cLchMTqchKhwLoj9ioTqSjjw2u4vUEREeg0FeFtf3gk/vLfD3VPLa3kpsoHB6XE7fRm3yxDmMq33xhhaZzjfvBQyJ0P2ft1Xt4iI9DoK8LZcYeAO3+HuGo8Ld3gESXF7MOlKxlg49EYtWiIiIntEAd7W/lc5tx246o7PGDQglv/9bKIfixIREdmeRqHvJq/Xsq68lpw+sYEuRURERAG+uzZW1dPo8ZLdR3OWi4hI4CnAd1NBWQ2AWuAiIhIUFOC7aV1ZLQDZKWqBi4hI4CnAd1NBWS3hbkNGUnSgSxEREVGA76515TVkpcTgdunyLxERCTwF+G4qKK1loLrPRUQkSCjAd4O1lrVlNQzUADYREQkSfg1wY8wxxpjlxpiVxpgbdnLcacYYa4wJihlTymoaqWlsZqAuIRMRkSDhtwA3xriB+4BjgRHAWcaY7ZbjMsbEA9cAs/1V266s1SVkIiISZPzZAp8MrLTWrrbWNgIzgJM6Oe6vwD+Bej/WtlNrWy4hUwtcRESChD8DfABQ2OZ5kW9bK2PMeCDLWvvWzl7IGHOpMSbfGJNfUlLS/ZV2UFBWi8tAZrIuIRMRkeAQNIPYjDEu4A7g17s61lr7oLV2orV2Ylpa2l6vbW1ZDRlJ0USGuff6e4mIiOwOfwZ4MZDV5nmmb1uLeGAU8KkxpgCYCswMhoFsa8tqNYBNRESCij8DfA4wxBiTa4yJAKYDM1t2WmsrrbWp1toca20OMAs40Vqb78caO6VLyEREJNj4LcCttR7gSuA9YCnwgrV2sTHmFmPMif6qo6sq65rYUtukSVxERCSohPnzzay1bwNvd9h20w6OPdQfNe1KyyImaoGLiEgwCZpBbMGqZRlRnQMXEZFgogDfhXXlLS1wBbiIiAQPBfguFJTWkB4fSUyEX882iIiI7JQCfBfWlusSMhERCT4K8F3QJWQiIhKMFOA7UdfYzKaqBl1CJiIiQUcBvhOtA9hS1QIXEZHgogDfidZLyNQCFxGRIKMA34mWSVy0DriIiAQbBfhOFJTVkBQTTmJMeKBLERERaUcBvhPrymvVfS4iIkFJAb4TBbqETEREgpQCfAcaPV6Kt9RpEhcREQlKCvAdKK6ow2u1CpmIiAQnBfgOaBUyEREJZgrwHdi2DrgCXEREgo8CfAcKymqIiXCTFhcZ6FJERES2owDfgbVltWSnxGCMCXQpIiIi21GA78DashrNwCYiIkFLAd6JZq+lsFyXkImISPBSgHdiY1U9jc1eXUImIiJBSwHeibWluoRMRESCmwK8EwW6hExERIKcArwTa8triHC76J8YHehSREREOuXXADfGHGOMWW6MWWmMuaGT/b8wxiw0xsw3xnxpjBnhz/parC2tJTMlGrdLl5CJiEhw8luAG2PcwH3AscAI4KxOAvpZa+1oa+1Y4F/AHf6qr6215bW6hExERIKaP1vgk4GV1trV1tpGYAZwUtsDrLVVbZ7GAtaP9bXUwNqyGrK1DriIiASxMD++1wCgsM3zImBKx4OMMVcA1wERwE/8U9o2pdWN1DY2k6MBbCIiEsSCbhCbtfY+a+0g4Hrgj50dY4y51BiTb4zJLykp6db3X9uyClmqutBFRCR4+TPAi4GsNs8zfdt2ZAZwcmc7rLUPWmsnWmsnpqWldV+FtLmETF3oIiISxPwZ4HOAIcaYXGNMBDAdmNn2AGPMkDZPjwdW+LE+ANaV1eAykJmsABcRkeDlt3Pg1lqPMeZK4D3ADTxqrV1sjLkFyLfWzgSuNMYcATQBW4Dz/VVfi4KyWjKSookIC7qzCyIiIq38OYgNa+3bwNsdtt3U5vE1/qynM7qETEREQoGamR2sLashWyPQRUQkyCnA26isbaKitkmXkImISNBTgLextrxlFTJ1oYuISHBTgLehVchERCRUKMDbWOebxEXTqIqISLBTgLdRUFZLenwkMRF+HZwvIiLSZUqqNq47ch/OmZId6DJERER2SQHeRkZSNBlJ0YEuQ0REZJfUhS4iIhKCFOAiIiIhSAEuIiISghTgIiIiIUgBLiIiEoIU4CIiIiFIAS4iIhKCFOAiIiIhSAEuIiISgoy1NtA17BFjTAmwthtfMhUo7cbX6630OXYPfY7dQ59j99Dn2D26+jkOtNamddwY8gHe3Ywx+dbaiYGuI9Tpc+we+hy7hz7H7qHPsXt01+eoLnQREZEQpAAXEREJQQrw7T0Y6AJ6CH2O3UOfY/fQ59g99Dl2j275HHUOXEREJASpBS4iIhKCFOAiIiIhSAEuIiISghTgIiIiIUgBLiIiEoIU4CIiIiFIAS4iIhKCFOAiIiIhKCzQBeyp1NRUm5OTE+gyRERE9oq5c+eWdrYaWcgHeE5ODvn5+YEuQ0REZK8wxnS6ZLa60EVEREKQAlxERCQEKcBFRERCkAJcREQkBCnARUREQpACXEREJAQpwEVEREKQAlxERCQEKcDbsNaytb4p0GWIiIjskgK8jd+9tIDj7v4i0GWIiIjskgK8jYykaIq21NHgaQ50KSIiIjulAG8jLy0Wa2FdWW2gSxEREdkpBXgbealxAKwqqQlwJSIiIjunAG8jJzUGgDWlCnAREQluCvA24qPCSYuPZE1pdaBLERER2SkFeAd5qbGsVhe6iIgEOQV4B3lpsepCFxGRoKcA7yA3NZaymkYqazWhi4iIBC8FeActI9FX6zy4iIgEMQV4B7lpsYBGoouISHBTgHeQlRyD22UU4CIiEtQU4B1EhLnITonRSHQREQlqCvBO5KbGslotcBERCWIK8E7kpsZSUFqD12sDXYqIiEinFOCdyEuLpa6pmY1V9YEuRUREpFMK8E7kpmokuoiIBDcFeCe2XQuuABcRkeCkAO9E34RIYiLcrC7RZC4iIhKcFOCdMMaQm6o50UVEJHgpwHdAAS4iIsFMAb4DeWlxFJbX0uBpDnQpIiIi21GA70BeaixeC4XltYEuRUREZDsK8B1ouZRMU6qKiEgwUoDvQMuqZLqUTEREgpECfAcSosJJjYtkjVrgIiIShBTgO5GnkegiIhKkFOA74axKpslcREQk+CjAdyIvLZbS6kYq65oCXYqIiEg7fg1wY8wxxpjlxpiVxpgbOtl/gTGmxBgz33e72J/1ddQyEr1A3egiIhJk/Bbgxhg3cB9wLDACOMsYM6KTQ5+31o713R72V32dyWsdia5udBERCS7+bIFPBlZaa1dbaxuBGcBJfnz/LstOicVl0Eh0EREJOv4M8AFAYZvnRb5tHZ1mjFlgjHnJGJPV2QsZYy41xuQbY/JLSkr2Rq0ARIS5yEqJ0bXgIiISdIJtENsbQI61dgzwAfBEZwdZax+01k601k5MS0vbqwXlpsZqNjYREQk6/gzwYqBtizrTt62VtbbMWtvge/owMMFPte1QXmoca0prsNYGuhQREZFW/gzwOcAQY0yuMSYCmA7MbHuAMaZ/m6cnAkv9WF+nctNiqWtqZlNVw64PFhER8ZMwf72RtdZjjLkSeA9wA49aaxcbY24B8q21M4GrjTEnAh6gHLjAX/XtSF7roibV9EuMCnA1IiIiDr8FOIC19m3g7Q7bbmrz+EbgRn/WtCt5bRY12X9waoCrERERcQTbILag0zc+iuhwt+ZEFxGRoKIA3wWXy5CTGsvqEk3mIiIiwUMBvhvy0rQqmYiIBBcFeFtlq2Dlh9ttzkuNpXBLHY0ebwCKEhER2Z4CvK0v/g2vXAodrvnOTY2l2WtZV14boMJERETaU4C3lTUFasugbGW7zXlpcQDqRhcRkaChAG8re6pzv25Wu825fZxLydZoVTIREQkSCvC2+gyB6GQobB/giTHh9ImN0JzoIiISNBTgbblcTjf6utnb7cpLi9WqZCIiEjQU4B1lTYGyFVBT1m5zbqouJRMRkeChAO+o5Tx4YftWeG5qHCVbG9ha3xSAokRERNpTgHeUMQ5c4dudB2+ZE12tcBERCQYK8I7CoyFj7HbnwVtWJVOAi4hIMFCAdyZrCqyfB55ta4Bn94nBZWCVRqKLiEgQUIB3JnsqNDfA+vmtmyLD3GQmx6gFLiIiQUEB3pmsKc59h/Pgzkh0TeYiIiKBpwDvTFw6pORtdx48NzWWNSU12A5zpYuIiPibAnxHsqY6l5K1CevB6XHUNDZTWF4XwMJEREQU4DuWPQVqS50lRn2m5qUA8PWq0kBVJSIiAijAdyyrZUKXbefBB6XF0Tchki9WKsBFRCSwFOA7kroPRCW1W5nMGMMBg1P5emUpXq/Og4uISOAowHekZWGTDlOqHjg4lS21TSzZUBWgwkRERBTgO5c9BUp/gNry1k0HDk4F4Ct1o4uISAApwHcma/uFTdITotinbxxfKsBFRCSAFOA7M2C8s7DJuvYTuhwwOJVv15RT39QcoMJERKS3U4DvTHg09N+30/PgDR4v363dEqDCRESkt1OA70r2VCj+rt3CJlPy+uB2GXWji4hIwCjAdyVrirOwyYbvWzfFRYYxLitJA9lERCRgFOC7ku0byNbhPPiBQ1JZUFxJZW1TAIoSEZHeTgG+K3HpkJzb6XlwazWtqoiIBIYCfHdkT3Va4G0WNtk3K4nYCLfOg4uISEAowHdHlm9hk/LVrZvC3S6m5vXReXAREQkIBfjuyJri3HdyPXhBWS2F5bUBKEpERHozBfjuSBsGUYntViYDOGiIM62qzoOLiIi/KcB3h8sFmZNhXfuBbIPT40iPj+TLlWUBKkxERHorBfjuyp4CpcvbLWxijOHAwal8peVFRUTEzxTgu6t1YZNv220+YHAq5TWNLN2o5UVFRMR//BrgxphjjDHLjTErjTE37OS404wx1hgz0Z/17dSACeAK2+48+AFaXlRERALAbwFujHED9wHHAiOAs4wxIzo5Lh64BpjdcV9ARcRAvzHbnQfvlxjFkPQ4nQcXERG/8mcLfDKw0lq72lrbCMwATurkuL8C/wTq/Vjb7smeCuvbL2wCLcuLltHg0fKiIiLiH/4M8AFAYZvnRb5trYwx44Esa+1bO3shY8ylxph8Y0x+SUlJ91e6I3mHgaceVn3SbvOBg1Opb/IyV8uLioiInwTNIDZjjAu4A/j1ro611j5orZ1orZ2Ylpa294trkXcoRCfDopfabZ6Sl4LbZXQeXERE/MafAV4MZLV5nunb1iIeGAV8aowpAKYCM4NqIFtYBIw4CZa9BY01rZvjo8IZm5Wk8+AiIuI3/gzwOcAQY0yuMSYCmA7MbNlpra201qZaa3OstTnALOBEa22+H2vctVGnQ1MtLH+n3eYDBqeysKhCy4uKiIhf+C3ArbUe4ErgPWAp8IK1drEx5hZjzIn+qmOPDdwf4vvDopfbbT5oSCpeC9+sVitcRET2vjB/vpm19m3g7Q7bbtrBsYf6o6Yuc7lh1Gkw+39Qt8U5Jw6MbV1etIRjRvULcJEiItLTBc0gtpAy6jTwNsHSN1o3hbtdTMnrw1c6Dy4iIn6gAP8xMsZBSh4sbD8a/YDBqawpraFoi5YXFRGRvUsB/mMY4wxmW/M5bN3Yuvmwoc4lbW8u2BCoykREpJdQgP9Yo08HLCx+tXVTXlocU3JTeHb2Oq1OJiIie5UC/MdKGwr9Rm/XjX7O1IGsK6/lC03qIiIie5ECfE+MOh2K86F8Teumo0f2pU9sBE/PWhvAwkREpKdTgO+JUac5922uCY8Mc3PGpCw+WrqJDZV1ASpMRER6OgX4nkjKgqyp203qcvbkbCww49vCzn9ORERkDynA99To02HzEti0uHVTVkoMh+yTxow562hq9gawOBER6akU4Htq5Clg3NsPZpsykE1VDXy0dHOAChMRkZ5MAb6nYlOdZUYXvQx226VjPxmWTkZiFM/M1mA2ERHpfgrw7jD6dKhYC0XbFk5zuwzTJ2fzxYpSCkprdvLDIiIiXacA7w7DpoE7Eha170Y/c1IWbpfh2W/XBagwERHpqRTg3SEqAfY52pmVzdvcurlvQhRHjejLi/mF1Dc17+QFREREukYB3l1Gnw7Vm6Dgi3abz5kykC21TbyzSPOji4hI91GAd5chR0FEPCx8sd3m/Qf1ITc1lmdmqRtdRES6jwK8u4RHw/BpsOQN8DS0bna5DGdPziZ/7RaWbawKYIEiItKTKMC70+jToaESlsxst/n0CZlEhLnUChcRkW6jAO9OeT+BtOHw+e3tBrMlx0YwbXR/Xp1XTE2DJ4AFiohIT6EA704uFxzyWyhdDktea7frnKnZVDd4eH3++sDUJiIiPYoCvLuNOBlSh8Jnt4N32zzo47OTGdYvnmdmr8W2mbFNRETkx9ijADfGxBljjjfGDOmugkKeyw2H/A5KlsLS11s3G2M4Z+pAFq+vYn5hReDqExGRHqFLAW6MedYYc7XvcTgwG3gDWGyMmbYX6gtNI0+B1H3gs3+1a4WfMm4A8VFh3PnhCrXCRURkj3S1BX4o8JXv8QlAPNAfuBn4U7dVFepcbjj4t84yo8veaN0cFxnGtUfsw+c/lGiVMhER2SNdDfAUYJPv8ZHAK9baTcCzwPDuLCzkjToN+gzerhV+3n4DGZQWy1/fWkKDR9OriojIj9PVAC8Bcn2PjwQ+8T2OAbyd/kRv1dIK37QIlr/dujnc7eKmE0aytqyWx74qCFx9IiIS0roa4C8CzxhjPgQSgA9828cCK7qxrp5h1OmQkgef/bPdWuGH7JPGEcPTueejFWyuqg9ggSIiEqq6GuC/A/4DLAKOtNbW+rZnAA91Y109gzvMaYVvXADL32m364/Hj6Cp2fLPd5cHqDgREQllXQpwa63HWnuHtfZaa+33bbb/n7X2we4vrwcYfQYk58Jnt7VrheekxvLzA3N5+bsi5q3bEsACRUQkFHX1MrJ9jTEj2zw/zhjzojHmZmNMWPeX1wO4w+Dg38CG7+GH99rtuvIng0mPj+TmN5bg9eqyMhER2X1d7UL/HzAawBiTCbwExAGXAH/r3tJ6kDFnQtLA7VrhcZFhXH/MML4vrOCVecUBLFBEREJNVwN8KDDP9/hUYI619ljgPODM7iysR3GHO63w9fNgxQftdp0ybgD7ZiXxz3eXUa2FTkREZDd1NcAjgJZh04cCLSOzfgD6dVNNPdO+Z0Fi9natcJfLcPMJIyjZ2sC9H68MYIEiIhJKuhrgy4HTjTHZONeBf+jb3h/QSKydcYfDQddB8VxY8X67XeOykzltfCaPfrmGgtKaABUoIiKhpKsB/hfg78Aa4Etrbb5v+1Fs61qXHRl7DqQMgrd/Aw3V7XZdf8xQwt2Gv721JEDFiYhIKOnqZWSvA9nABOD4Nrs+An7bjXX1TGERcNK9UFEIH/2l3a70hCiuOnwIHy7dzGc/lASoQBERCRVdXk7UWrvJWjsfiDDGRPm2fWOtVdNxdwzcH6ZcBt8+CAVfttt14QE55KXGcuPLC9hS0xigAkVEJBR0OcCNMRcaY1YC1UC1MWaFMeaC3fzZY4wxy40xK40xN3Sy/xfGmIXGmPnGmC+NMSO6Wl9IOPwmSM6B16+ExtrWzZFhbu6aPo7S6kaue2G+rg0XEZEd6upELtcA9wMzgdN8tzeB+40xV+3iZ93AfcCxwAjgrE4C+llr7Whr7VjgX8AdXakvZETEwon3wpY18PFf2+0anZnIn04YwSfLS/jv56sCVKCIiAS7rrbArwKusdZeZ6193Xf7FfAr4Jpd/OxkYKW1drW1thGYAZzU9gBrbVWbp7FAz22C5h4Eky6GWQ/Aulntdp07JZtpY/rzf+8tZ9bqsgAVKCIiwayrAZ6FM2Cto498+3ZmAFDY5nmRb1s7xpgrjDGrcFrgV3f2QsaYS40x+caY/JKSEB7wdcRfICkLXr8CmupaNxtjuO20MeT0ieXq5+ZRsrUhgEWKiEgw6mqAF+FM4NLRob59e8xae5+1dhBwPfDHHRzzoLV2orV2YlpaWne8bWBExsGJ90DZSvjk1na74iLDuO+c8VTWNXHt8/No1vlwERFpo6sB/gBwtzHmH76FTI4zxtwG3IVzbnxnimnfSs/0bduRGcDJXawv9OQdChMuhG/ug8I57XYN75/AX08axVcry7j7Iy23LiIi23T1OvD/w1kT/BycwWtvAmcDv7HW/nsXPz4HGGKMyTXGRADTcQbDtTLGDGnz9Higd6TWkbdAfAa8fjk01bfb9dOJmZw2PpO7P17BFytC+HSBiIh0qx9zHfh91tpsIBFItNZmW2sf2I2f8wBXAu8BS4EXrLWLjTG3GGNO9B12pTFmsTFmPnAdcH5X6wtJUQlw4t1Q+oMzV3obxhj+evJIhqTHce2M+WysrN/Bi4iISG9irN35uVVjzPs7PaANa+1Re1xRF02cONHm5+fv+sBQ8PqVMP8ZuPhDGDCh3a6Vm7dy4r1fMTIjgecumUqYu8vfvUREJAQZY+Zaayd23L47KVDchZvsiaNvhfj+8OKFUNP+8rHB6fH849TRzCnYwu3vLQ9QgSIiEizCdnWAtfZCfxQiQFQinPEUPH4cPH8unPcahEW27j5p7ADmFJTzv89XMyA5mvP2ywlYqSIiEljqhw02mRPgpPtg3dfw5q/arR0OcPMJIzlieF/+PHMxb3y/PkBFiohIoCnAg9Ho0+GQG5zz4V/f3W5XmNvFvWePY+LAZK57Yb5GpouI9FIK8GB1yPUw8hT44M+w7O12u6LC3Tx8/iQGpcVx2VNz+b6wIjA1iohIwCjAg5XLBSc/ABnj4OWLYcOCdrsTo8N58ueTSYmN4MLH57CqpDpAhYqISCAowINZeDSc9ZwzuO25s2Drpna70xOieOqiKbgMnPfIt7pGXESkF1GAB7v4fnD2DKgrhxlnt1v0BCA3NZbHL5xMZV0T5z06m4raxgAVKiIi/qQADwX994VT/gfF+c5kLx1Gpo8akMiD502goLSWnz8+h7rG5gAVKiIi/qIADxUjToTDb4JFL8Fn/9xu9/6DUrn7rLHML6zgl8/MpdHjDUCRIiLiLwrwUHLgdbDvWfDpP2DW9tPPHzOqP7eeMppPl5fwy6fn0uBRS1xEpKdSgIcSY5z1w4dNg3dvgDmPbHfIWZOz+dvJo/ho2WZ+8dRc6psU4iIiPZECPNS4w+H0x2DI0fDWdTDv6e0OOXfqQP5+ymg+WV7CZQpxEZEeSQEeisIi4IwnIe9QZ1Dbghe3O+TsKdn887TRfL6ihEuezFeIi4j0MArwUBUeBdOfg4EHwKuXwZLXtzvkzEnZ/PO0MXy5spRLnszX6HQRkR5EAR7KImLg7OchcyK89HNY/s52h5wxMYvbT9+XL1eWcvGTusRMRKSnUICHusg4OOdF6DcaXjgPVn603SGnT8jk3z/dl29WlfHzx+dQ2+gJQKEiItKdFOA9QVQinPsKpA51Zmtb88V2h5w6PpM7zhjL7DVlXPjYHKobFOIiIqFMAd5TxKTAea9Bcg4881NY/Op2h5w8bgB3njmW/LVbOPN/37C5SnOni4iEKgV4TxKbCue/4XSnv3gBfHwreNvPyHbS2AE8fP5E1pTWcMr9X7Nys1YxExEJRQrwniYuHS54E8adC5//C174GTRsbXfIYUPTef7S/WjweDntga+ZU1AeoGJFROTHUoD3RGGRcOK9cMw/nZHpDx8J5WvaHTI6M5FXL9+fPrERnPPwbN5ZuCFAxYqIyI+hAO+pjIGpv4BzX4atG+Chw2D1Z+0OyUqJ4eVf7s/oAYlc/ux3PPbVmh28mIiIBBsFeE836DC49BOI6wtPnQKz/9duOdLk2AieuXgKR43oy1/eWMKtby3B67U7eUEREQkGCvDeICUPLvoA9jka3vkdzLwKPA2tu6PC3dx/zgTO328gD32xhqtnzNPUqyIiQU4B3ltEJcCZz8DBv4V5T8ETJ0J1Setut8tw84kjufHYYby5YAOnPfA1a0prAliwiIjsjAK8N3G54Cd/hNMfhQ3fw4OHOvc+xhguO2QQD583keKKOqbd/QWvzisKXL0iIrJDCvDeaNRp8PN3AQuPHL3dpC9HjOjLO9ccxMiMRH71/Pdc98J8ajRzm4hIUFGA91YZY+HST6H/mE4nfemfGM2zl0zhmsOH8Nq8Yk6450sWr68MVLUiItKBArw3i0t3Zm4b23bSl20zs4W5XfzqyH145uKp1DR6OOW+r3ni6wKs1Sh1EZFAU4D3dmGRcNK9cMxtsPxteOQo2FLQ7pD9BvXhnWsO5qAhqfx55mIufWouW2oaA1OviIgACnAB36Qvv4RzXoKqInjwsO3WFk+JjeDh8ydy07QRfLp8M4ff8Rkv5hfqmnERkQBRgMs2gw+HSz6B+P7w3HRnffGtG1t3G2P4+YG5zLzyQHJTY/ntSws488FvWLaxKoBFi4j0Tgpwaa/PIGdw2+E3wfJ34d7JMOeRdgPchvdP4MXL9uNfp41h5eZqjr/7S/725hKtMS4i4kcm1AckTZw40ebn5we6jJ6pbBW8eS2s+RyypsAJd0H68HaHbKlp5F/vLWfGnHWkx0dy07SRHDe6H8aYwNQsItLDGGPmWmsndtyuFrjsWJ9BcN5MOPm/ULoC/nsQfPw3aKpvPSQ5NoJ/nDqal3+5P6lxkVzx7Hec9+i3msVNRGQv82uAG2OOMcYsN8asNMbc0Mn+64wxS4wxC4wxHxljBvqzPumEMTD2LLgyH0afDp/fDg/sv93KZuOzk3n9igO4+YQRzF9XwdF3fs6/319OXaPmVBcR2Rv8FuDGGDdwH3AsMAI4yxgzosNh84CJ1toxwEvAv/xVn+xCbB845b/ws9fAeuHJE+Gln0PVtnXEw9wuLjggl49+cwjHj+nPPR+v5Mg7P+ODJZsCV7eISA/lzxb4ZGCltXa1tbYRmAGc1PYAa+0n1tpa39NZQKYf65PdMegwuPwbOPRGWPom3DsRvr4XmptaD0mPj+LOM8cy49KpxES4ueTJfC56fA7rymp38sIiItIV/gzwAUBhm+dFvm07chHwTmc7jDGXGmPyjTH5JSUlnR0ie1N4NBx6A1wxCwbuD+//Af53MBR81e6wqXl9eOvqg/jDccOZtbqMI+/8jLs+XKGlSkVEukFQDmIzxpwLTARu72y/tfZBa+1Ea+3EtLQ0/xYn26TkwdkvwPRnnSlYHz8OXrkMqje3HhLudnHJwXl89OtDOWJEX+788AeO/s/nfLR0k6ZkFRHZA/4M8GIgq83zTN+2dowxRwB/AE601jb4qTb5sYyBYcfDFbPhoF/DopfhngnwzX3tRqv3S4zivrPH8/RFU3C7DBc9kc+J937F2ws30KzZ3EREusxv14EbY8KAH4DDcYJ7DnC2tXZxm2PG4QxeO8Zau2J3XlfXgQeZ0pXwzm9h1cfOjG4H/RrGn+fMue7T6PHyyndF/PezVRSU1ZKXGstlh+RxyrhMIsKCslNIRCRgdnQduF8ncjHGHAf8B3ADj1prbzXG3ALkW2tnGmM+BEYDLUOb11lrT9zZayrAg9Saz50lSgtnQWIWHPxbGHs2uMNbD2n2Wt5ZtIEHPl3F4vVV9EuI4uKDcjlrcjaxkWEBLF5EJHgERYDvDQrwIGat0xL/5FYongvJOXDI9TD6DHCHtTnM8vmKUh74dCWzVpeTFBPO+fvlcPFBucRHhe/49UVEegEFuASOtfDDe06Qb1wAfQbDITfAqFPB5W536HfrtvDAp6v4YMkm+sRGcM0RQzhrcjbhbnWti0jvpACXwLMWlr0Jn/wDNi+GtGHO5WjDTwJX+4BeUFTB399eyqzV5eSlxnL9scM4akRfzbEuIr2OAlyCh9cLS16DT2+D0uWQPhIOuxGGTXNGtftYa/lo6Wb+8c5SVpXUMDknhRuPG8a47OTA1S4i4mcKcAk+3mZY9Ap8dhuUrYR+Y+Cw38M+x7QLck+zl+fzC7nzgx8orW5k2pj+XH/MMLJSYgJYvIiIfyjAJXg1e2Dhi06QbymAjHFw2B9g8BHtgry6wcODn63iwS9W4/XCtDH9OXtKNhMGJqtrXUR6LAW4BL/mJvh+Bnz2L6hcB6lDYfIlsO9ZEBnXetjGynru/3Qlr3xXTHWDh336xnH25GxOGZdJYoxGrYtIz6IAl9DhaYRFL8Hs/8GG+RCZAGPPccK8z6DWw2oaPLy5YD3Pzl7H90WVRIW7OH50BmdPyWZ8dpJa5SLSIyjAJfRYC0VznCBf8hp4PTD4SJhyGQw6vN3I9UXFlTz77Tpen1dMTWMzw/rFc+EBOZrdTURCngJcQtvWjZD/GMx9DKo3QcogmPILGHcORMS2HlbT4GHm9+t5etZaFq+vIiMxil8cOogzJmYRFe7eyRuIiAQnBbj0DJ5GWDoTZj0AxfkQlQSTLoLJl0J8v9bDWmZ3u+ejFeSv3UJafCSXHZzH2VOyiYnQNK0iEjoU4NKzWAuFs+Hre2DZW84c66PPgP2ugL4j2hxmmbW6nHs/WcFXK8tIiY3gogNzOW+/gZqmVURCggJceq6yVU6LfN7T4Klzzo/vfyXkHdbuMrS5a7dw3ycr+XjZZhKiwjhzUhZHj+zHuOxk3C4NeBOR4KQAl56vthzyH4HZD0LNZmfO9X2nw5jpkLRtKfpFxZXc98lKPly6iaZmS2pcBIcP68tRI/tywOBUnSsXkaCiAJfew9MAi152WuRrvwIM5B4E+54NI05sHfRWVd/EZ8tLeH/JJj5dtpmtDR6iw90csk8aR47oy+HD00mKiQjs7yIivZ4CXHqn8jWw4Hn4/jlnlrfwWBhxEow9CwYe2HopWqPHy6zVZby/ZCMfLNnEpqoGwlyGQ/ZJ48SxGRw5oq8Gv4lIQCjApXezFtbNgu+fhcWvQUMVJGTCyJNh1GnO9K2+8+Ver2VBcSXvLNzAzO/Xs6GynpgIN0eN6MtJYwdw4JBULW8qIn6jABdp0VTnjFxf+BKs/BC8TZCc6wT5qNPajWL3ei3fFpTz+vz1vL1wA5V1TaTERnD86P6csG8G47OTCFOYi8hepAAX6UzdFlj6pnPOfM1nYL2QNhxGnQojT4XUwa2HNnia+fyHUl6fX8yHSzdR3+QlPiqM/fL6cNCQVA4ckkZOnxhN4Soi3UoBLrIr1SXOlK2LX/UNfgPSR8DwE2H4CdB3ZGs3e3WDh0+Xb+bLFaV8saKU4oo6AAYkRXPg4FQOHJLKAYNTSYnVIDgR2TMKcJGuqCyGpW84s76t/RqwkJLnC/MTYcD41jC31rK2rJYvVpby1YpSvl5VSlW9B2PgwMGpTJ+UzZEj+mpOdhH5URTgIj9W9WbnnPnSmbDmc2dRlYRMGHqsE+R9R0HaMAhzWtueZi8Liyv5eNlmXp5bxPrKelJiIzh13ADOnJTFkL7xAf6FRCSUKMBFukPdFlj+rtM6X/WxM/MbgCvMWb+83ygn0PuNgn5jaI7uwxcrSnh+TiEfLNmEx2uZMDCZMydlMW1Mf12aJiK7pAAX6W7eZmca100LYeMi2LTIud+6ftsxGeOdAXEjTqY0LJ1XvitixpxCVpfUEBcZxgGD+zBhYDITBiYzMiNRs8CJyHYU4CL+UlPmhHrxXFgyEzbMd7ZnToZRp2KHn0j+lmheyi9i1poy1pbVAhDhdjFqQEJroI8fmEx6fFTgfg8RCQoKcJFAKVvljG5f9KoT7BjI3g9GngK5B1MSmcV3RVv5bu0W5q7dwoLiSho9XgCyU2KYODCZiTkpTMpJZlBaHC4tvCLSqyjARYJB6QrnMrXFr8LmJc62sChnEFy/UdB3NI1pI1jqzebbDV7mrt1C/tpySqsbAUiKCWfiwGQmDHQCfXRmIpFh6nYX6ckU4CLBpnSF082+ceG28+e1pdv2J2ZB2jBsSh5lUVksbkjj6y2JfLQ+gpWl9QAkRIVx7tSBXLB/DukJ6m4X6YkU4CLBzlqo3uQbEOcbGFe6HMpWQ1PNtuPcEXgSB1Iemcn3dal8WJJEgRnA0JETOOewcQztp8vURHoSBbhIqGoJ9rKVzvn08lW++9XOfXND66FlNp6SqBwSs0bQL280Jn0EDDwAwtU6FwlVOwpwXYQqEuyMgfh+zi3nwPb7vM1QsQ5KV1C3YSkbl82jccMy0la8jVn5PAAedwxl/Q6iPOsItmb9BBObQrjbRbjbEBXuJjslRquriYQgtcBFepgGTzOvz1/PC5/NJ65sAUe45nKkey59TQUe6+Jb7zDe907kg+YJFJNGVLiLkRmJ7JuZxL5ZiYzNSiI7RYuyiAQLdaGL9DLWWtaV11Lf5KXJ4yFs03zi17xHUuGHxFauAKAyfgibXWlsaoigqDaMCm80W200nog4+iSn0jc9jQH9+jEwoy+pfVIxUYkQGQ9hkQH+7UR6D3Whi/QyxhgG9ondtiHzMJhwmPO4bBUse4vE1Z+SWFvGEPdGrHsrtr4KV3MDWKDcd1u2/Ws3uyIgMh5XVAKmzyCna3/ggZAxFtzhe/+XExG1wEWkA08DNGyFhirqqrewbv0m1m/axKbNJZRvKaNu6xZibC3x1JLirmNMeBGZnnUA2PBYTPZUyDkAcg6CjHEKdJE9pBa4iOyesEjnFptKdAoMzYahbXY3eJpZsamaxesr+bq4kvsLK9i0vpCJZhn7NS/h4DXLyVn1EQDe8BhMv9GY2DSITYO4dOc+NtV3nw7RSUDL+XZfg6JjwyIs0pnwJiyydRlXkd5OLXAR2WM1DR6+L6wg3zcd7Jp1axnRuIipriUMdxeT5qoihSoSbBUu9uRvjtkW5OHRzuPwaEgf7kxPmzUF+o4El2ank55Dg9hExG+8XssPm7eSX7CFVSXVlNc0Ul7TSEV1Hc01ZbhqS0nwVpBKFQmmBhfQLzGKgX1iyOkTx8DUGOIifV3v1gvNTc7SrU314PHdmuqc7v7Galg/f9sqcJEJkDnJCfTsKTBgAkTE7qhUkaAXFF3oxphjgLsAN/Cwtfa2DvsPBv4DjAGmW2tf8md9ItI9XC7DsH4JDOuX0Ol+ay01jc1sqWlkbVktcwrK+bKgnLsLtlC/wlnIJS81lsm5KUwYmMyAPtGkxUWSFh9JYnT49pe4WQuVhbBuFqz7BtbNhk9uBayzVntSNsT1c7rw4333cX23bYvr63Trq+UuIcRvLXBjjBv4ATgSKALmAGdZa5e0OSYHSAB+A8zcnQBXC1yk52j0eFm0vpI5a8r5dk05cwrKqar3tDsm3G1IjYsk1RfoaXGR5KXFMrRfPMP7J5AeH+kEfN0WKJwDhbNgSwFs3eTMaFe9CRqqOnl34zs3nw5xvvPzLefs4/tD8kBIznHCXufhxY+CoQU+GVhprV3tK2gGcBLQGuDW2gLfPq8f6xKRIBER5mJ8djLjs5O57JBBeL2WNWU1bKqqp7S6kZKtDZRWN7Teb6qqZ0FRJc/nb5tONjkm3Gn9949neL/hDBs6mX36xhMV3qZ13VgLNZvbh3pNCVRvdm41m6F8trOtqbZ9kWFRkOQL8+QcJ9iTsiE6BaKTnUF5UUnOuXkFvexF/gzwAUBhm+dFwJQf80LGmEuBSwGys7P3vDIRCUoul2FQWhyD0uJ2elxFbSPLNm5l2YYqlm/aytINW5nxbSF1Tc0ARLhd7JuVyKScFCblpDB+YDKJLQG8Kw3VsHUDbFkLW9ZAxVqnRb+lwOmu77Q1D7gjnCCPTnKCPSHDWWEuKdu5JWZBUpYzMU5HzU1OD0LLrbbcmc8+fYR6AKRVSF5GZq19EHgQnC70AJcjIgGWFBPB1Lw+TM3r07rN63Vmolu6oYp5hRV8u6acBz9fzf2frsIYGNYvgck5yUzMSWFsVhIZSdG4XZ0EY2QcRA6B1CHb77PWCdiKdc59fQXUVbR/XF/hBPCG72HZW9Dc2P41opOdMDcGan2B3bh1x79sdIoT5H1HOKPv00dC+jCISuz6BychzZ8BXgxktXme6dsmItLtXC5DTmosOamxHDu6PwC1jR7mF1YwZ80W5hSU8+LcIp74Zi3gdN/n9IkhNzWW3NQ48lJjyU2LJTc1lj6xERhjsNbS7LV4fLfmZkuTNxYShpPUN5ywXS0K4/U63fMV67bdKguhotAJ8LThvm74ZIhJ2fY4OtkZbb95KWxaDJuXwPxnnW0tYtOc0fbhsRARA+ExvucxzvOI+G1d/DEpzheBtvfhMXunZd/cBLVlUFPqXD0Qlbjtpil594g/A3wOMMQYk4sT3NOBs/34/iLSy8VEhLH/oFT2H5QKgKfZy5INVSxZX8Wa0hpWl9awcnM1Hy/bTFPzts69CLeLZl9474gxkBIT4Qys8w2uS4t3BtulJ0QyKC2OwelxRLWsLJc1ueu/QO7B2x5b63wB2LwUNi92uvibap3z+001zn1t2bZtjdXtA78jd4QzYn9nwiIhIs65RcY5XxDaPm8b1rWlzn19xU5eL8o5zdAS6NHJEN8X4jOczyjBdx+fATF9wLWDL0jNHqdnw9vkXEbYS04x+PU6cGPMcTiXibmBR621txpjbgHyrbUzjTGTgFeBZKAe2GitHbmz19QodBHpbp5mL8UVdawurWFNSQ2bttYT5jK4XS7CXIYwt3HuXS7C3E5YlFU3UuIbYNd6q26g0bNtTK7bZchNjWWYb8T88P7xDOuXQP/EKP+s/uZp9J1XL3e69VvvfV33tnnHP2ut77r7GqeLv6Ha99j3xaCh2rkMLzbNCduW2fZiUiG2j3MfHg31VU6o11e2uffdasth60Zn8GDHCX9c4c7rATQ3OF8Wmhudm20z7jkqEfqPdabxzRjnzM+fNLDroe5tdn7f5gbn3tPgey+7rVfE7Z82sCZyERHxM2stWxs8bKqsZ8XmapZuqGLphq0s21hF0Za61uMSosJIjY8k3OUiPMz5YhDh3vY43O0iPiqMvglR9EuIpF9iFH0TnFt6fOSuu+5DTXOTc2XA1o1Qtd6537rBuULAGKe3wB0BYRHbHrvDwbihfBWsnwebljgtcnDCtiXUo5Pbf3loGafQ8rix2gnrnX2ZaRGZ6Jx+aD0V0cd5PP58Z1xCN1GAi4gEkar6Jn7YuJWlvtHzlXVNNDV7aWq2vnvnsafZS4PHy9Z6D5u31rfr2gcnz1LjIukTG0G424XbZQh3G9xtegjCXIbIMDfpCZFkJEbTPymKjKRoMhKjSYuP7HzwXqjzNDjjBdbPc24b5junG7weMC5ft32Sr+s+advzluVy3ZG+LwiR29YHcPvm4m+5MqC2bFsvRtvH05+BvEO77VdRgIuIhDiv11Je28jGyno2b61nY2UDG6vq2VxVT3lNY+vgOk+zt/W+2WtparbUNzWzsaqe2sb2Lcswl6FvQhQZSVGMzEhkXHYS47OTyUyO9k+3vj811Tvd4JHxe/c8ubXd+vrBMJGLiIjsAZdr2yx00PXLxqy1VNV5WF9Zx4bKOtZX1LOhso4NFfUUbqnl+TmFPP51AeC06sdnJzEuO5nx2UmMyUwiOsK93et5LTR7LV5riXC7cAVzaz48yrntbX764qMAFxHpJYwxJMaEkxgTzvD+289T72n2smzjVuYVVjBv7Ra+W7eF95dsApwBeNHhbpq9zmj8zkblx0a4GZmRyKgBiYzJdO7zUmODO9RDmLrQRURkh8prGpm3bgvzCyuoaWgmzG1wGYPbBW6XC7fvsctl2FRZz8LiShavr6LBN/q+JdRHZyaSmxrrOz/vcn7GtJyrd14zPMxFUnQ4KbERJMVEkBAV1vO68X8EdaGLiEiXpcRGcPjwvhw+vO9u/4yn2cvKkmoWFlWyqLiSBcWVPD1rbWuo7y63y5AUHU5ybATJMeEkx0TQPzGKAcnRziC8pGgyk6JJjYvsla18BbiIiHSrMLerdTnZn050JuD0NHsprW50ut6bt3XBe63F0+zcN3i8VNY1Ul7TREVtI1tq2z9eU1rD16vKqG7YfoW6/onRZCRFERvhtNpdxvkC4DIGY5zWvstAQnQ4GUnRDPB9ARiQFLoj8RXgIiKy14W5XfRL7J4BZFX1TRRvqWN9hXMrqnAG5K2vqGPT1nq8XvBa67s5j61vsF1FbeN2S9SGuQz9EqMYkBRN/8SobbPpxUeSHh/VOrNeUkwna9EHkAJcRERCSkJUOAn9Ox+Itzu21je1Bn5xxbYvAusr6slfu4WSrQ2ddveHuw0psRHER4UTHxVGXGQYCW0ex0eFExcVxlEj+pKVErOnv+YuKcBFRKRXiY8KZ2i/cIb262QpV5zL46obPO2mxN1c5dyXVTewtd5DdYOHqnoPxRV1VNd72FrvaV2+dp++cQpwERERfzPG+FrZ4eTtYi36tjzNXqobPNtdL7+3KMBFRES6QZjbRVJMhN/er4fNgC8iItI7KMBFRERCkAJcREQkBCnARUREQpACXEREJAQpwEVEREKQAlxERCQEKcBFRERCkAJcREQkBBlrbaBr2CPGmBJgbTe+ZCpQ2o2v11vpc+we+hy7hz7H7qHPsXt09XMcaK1N67gx5AO8uxlj8q21EwNdR6jT59g99Dl2D32O3UOfY/fors9RXegiIiIhSAEuIiISghTg23sw0AX0EPocu4c+x+6hz7F76HPsHt3yOeocuIiISAhSC1xERCQEKcBFRERCkAK8DWPMMcaY5caYlcaYGwJdT6gwxjxqjNlsjFnUZluKMeYDY8wK331yIGsMdsaYLGPMJ8aYJcaYxcaYa3zb9Tl2gTEmyhjzrTHme9/n+Bff9lxjzGzfv+3njTERga41FBhj3MaYecaYN33P9Tl2kTGmwBiz0Bgz3xiT79vWLf+uFeA+xhg3cB9wLDACOMsYMyKwVYWMx4FjOmy7AfjIWjsE+Mj3XHbMA/zaWjsCmApc4fv/T59j1zQAP7HW7guMBY4xxkwF/gncaa0dDGwBLgpciSHlGmBpm+f6HH+cw6y1Y9tc+90t/64V4NtMBlZaa1dbaxuBGcBJAa4pJFhrPwfKO2w+CXjC9/gJ4GR/1hRqrLUbrLXf+R5vxfmjOQB9jl1iHdW+p+G+mwV+Arzk267PcTcYYzKB44GHfc8N+hy7S7f8u1aAbzMAKGzzvMi3TX6cvtbaDb7HG4G+gSwmlBhjcoBxwGz0OXaZr9t3PrAZ+ABYBVRYaz2+Q/Rve/f8B/gd4PU974M+xx/DAu8bY+YaYy71beuWf9dh3VGdyM5Ya60xRtcr7gZjTBzwMnCttbbKafQ49DnuHmttMzDWGJMEvAoMC2xFoccYMw3YbK2da4w5NMDlhLoDrbXFxph04ANjzLK2O/fk37Va4NsUA1ltnmf6tsmPs8kY0x/Ad785wPUEPWNMOE54P2OtfcW3WZ/jj2StrQA+AfYDkowxLQ0W/dvetQOAE40xBTinE38C3IU+xy6z1hb77jfjfKGcTDf9u1aAbzMHGOIbZRkBTAdmBrimUDYTON/3+Hzg9QDWEvR85xcfAZZaa+9os0ufYxcYY9J8LW+MMdHAkTjjCT4BTvcdps9xF6y1N1prM621OTh/Cz+21p6DPscuMcbEGmPiWx4DRwGL6KZ/15qJrQ1jzHE4533cwKPW2lsDW1FoMMY8BxyKs0TeJuDPwGvAC0A2znKvZ1hrOw50Ex9jzIHAF8BCtp1z/D3OeXB9jrvJGDMGZ1CQG6eB8oK19hZjTB5OSzIFmAeca61tCFylocPXhf4ba+00fY5d4/u8XvU9DQOetdbeaozpQzf8u1aAi4iIhCB1oYuIiIQgBbiIiEgIUoCLiIiEIAW4iIhICFKAi4iIhCAFuIjsdcaYQ40x1je/toh0AwW4iIhICFKAi4iIhCAFuEgvYIy5yhizzBhTb4xZYYz5Q8uc1saYAmPMrcaYh40xVcaYUmPM340xrjY/H2+M+Z8xpsQY02CMyTfGHNXhPdKNMY8ZYzb53me5MebnHUoZboz53BhTa4xZYow51g+/vkiPpNXIRHo4Y8zNwIXAtcB8YDjwXyAK+JPvsKtwphGehLPYwn9xpsW9y7f/Ud++c4F1wC+AN40xY6y1y3zzjn8G1AHnAKuBwThTbrb1f8D1OEt8/h543hgz0Fq7pRt/ZZFeQVOpivRgxpgYoBQ41Vr7bpvt5wF3W2uTfCtOFVprD2qz/+/Az6y1WcaYwcAK4Hhr7dttjvkOmG+t/bkx5iLgPmCwtbaokzoOxVkI47SWldaMMX1x1kI+xlr7Xjf/6iI9nlrgIj3bSCAaeLnDmsNuIMoYk+Z7/k2Hn/sKuNEYkwCM8G37vMMxn+Ms1QkwAVjSWXh3ML/lgbV2kzGmGei7O7+IiLSnABfp2VrOY/8U+KGT/f5e2ayxk20aiyPyI+gfjkjPthioB/KstSs7uTX7jpva4ef2B4qttVW+1wA4uMMxB+OsbQwwFxih67xF/EcBLtKDWWurgb8DfzfGXGGMGWqMGWmMmW6M+WebQ8caY242xuxjjDkbuAb4t+81VgEvAvcbY442xgwzxtwFjAJu9/38czjrGs80xhxhjMk1xhxujDnTX7+rSG+jLnSRHs5a+1djzAbgSpxQrsPpTn+8zWH3AAOBfKAJuJdtI9ABLsYJ66eBBGAhMM1au8z3HrXGmEOAfwEzgDigALhtb/1eIr2dRqGL9HK+UegPW2v/FuhaRGT3qQtdREQkBCnARUREQpC60EVEREKQWuAiIiIhSAEuIiISghTgIiIiIUgBLiIiEoIU4CIiIiHo/wHgtZYnZGVPvwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 504x576 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sg.utils.plot_history(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(\n",
    "    train_gen, epochs=50, validation_data=val_gen, verbose=1, shuffle=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sg.utils.plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Threshold identification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_gen = generator.flow(test.index, test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "94/94 [==============================] - 391s 4s/step - loss: 0.0933 - acc: 0.8795\n",
      "\n",
      "Test Set Metrics:\n",
      "\tloss: 0.0933\n",
      "\tacc: 0.8795\n"
     ]
    }
   ],
   "source": [
    "test_metrics = model.evaluate(test_gen)\n",
    "print(\"\\nTest Set Metrics:\")\n",
    "for name, val in zip(model.metrics_names, test_metrics):\n",
    "    print(\"\\t{}: {:0.4f}\".format(name, val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predictions = pd.DataFrame(model.predict(test_gen), index=test.index, columns=test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results = pd.concat({\n",
    "    \"target\": test, \n",
    "    \"preds\": test_predictions\n",
    "}, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAduElEQVR4nO3de3hVd53v8fc393so7CRQSAmXAIVebVqpJYzTqqXqtH1sVTqjp94Gq9KLOmo9M9PnmTrn8XZG22qPtVo9jjda69GDlbbjsSrQFgq1FwWyIQFaoOyQBEh2Qu75nT/2TtikgezC3lnZa31ez5PH7L1/sL7L0A+L7++31s+cc4iISObL8roAERFJDQW6iIhPKNBFRHxCgS4i4hMKdBERn8jx6sChUMjV1NR4dXgRkYz0/PPPtzrnKsb6zLNAr6mpYevWrV4dXkQkI5nZKyf7TC0XERGfUKCLiPiEAl1ExCcU6CIiPpFUoJvZCjMLm1mjmd05xuffNLMX4187zexoyisVEZFTGneVi5llA/cDbwf2A1vMbK1zbvvwGOfcpxPG3wpcnIZaRUTkFJK5Qr8MaHTO7XbO9QFrgOtOMf4m4OepKE5ERJKXzDr0mcC+hNf7gTePNdDMZgNzgKdO8vkqYBXAOeec84YKFX86eqyPxkOdNB7qpLWzl4rSfKaXFzK9rIDp5QWUFeRgZl6XKZIRUn1j0UrgUefc4FgfOuceBB4EqKur04PYA8I5R6SjZyS4h7+aWjpp7ew75a8tysseCffp5QVMLytgRnkBVWUFzCgvZHp5AdOK88jKUuiLJBPoB4DqhNez4u+NZSXwqTMtSjLTwOAQrxw+RtOhThpb4qF9qJOmli46ewdGxpUV5DC/soQrF1Uyv7Ik9lVRSmVZPi3RXiIdPUTaY18H23to7ujhYHs3m3cfprmjh4GhE68FcrONytJ40JcXMCPhL4AZ5QVMLy+ksjSf3Gwt6hJ/SybQtwC1ZjaHWJCvBP5+9CAzWwScBTyb0gpl0unuG6SpJXaFnXjFvbeti/7B42E7vayA+ZUl3HjJLOZVFDMvHt4VJfknbaNUTy2iemrRSY89OORo64yF/sF46Cf+BbD9tQ5+v6OZnv6hE36dGYRK8keu9o9f5R+/8p9eXkBRnmdPwxA5Y+P+6XXODZjZauBJIBv4gXNum5ndDWx1zq2ND10JrHHa0843EvvbjQlX3QeOdjP8U84ymD2tmHkVJVx1btXIFfe8imJKC3JTXlN2llFZVkBlWQEXzBp7jHOOju4BDnZ0x67w208M/1fbjvHcnsO0d/e/7teWF+Yeb/Ekhn/8f2eUFVJWqL6+TE7mVf7W1dU5PZzLe845Drb3nBDaTWP0t/NzsphbMdweKRkJ7ppQEfk52R6ewek71jdwwhX+8fbO8fBv7exl9H8iBblZsf79qNbOyBV/WQGhknz19SUtzOx551zdWJ/p35cBMdzfHpmQTAjvrr7jc9jlhbnMryzhqkVVzKssHulvzzyrkGyfBVRRXg5zK0qYW1Fy0jH9g0McivYSae8+HvTtPRzsiF35P7fnMIeiPSe0mgBysoyqsgKqyvJHJm9Ht3uqygrIy1FfX1JHge4zb7S//d666lhvO37VHSrJUzshQW52FjOnFDJzSuFJxwwNOdq6+hKu9uPhH7/y33Gwg6caDtHd//rFX6GSPGaeVcTSOVOpr62gruYsCnIz81884j21XDLUka6+kZ524teBo90jYxL72yOrSdLY35aTc87R0TOQ0NLpJtLeS6Sjm6aWLl549Qj9g478nCwumzOV+toQy+ZXcO6MUv0FKyc4VctFgT6JjdXfHm6XtHWd2N9ODO3h7zO5vx00x/oG2LznMBt2trKxsYWdzZ1A7Ap+2fwQy2orqK8NUVVW4HGl4jUFegbZcbCD763ffcr+9vxRV9wzpxRqAs5nIu09bGxsZeOuFjY2to5MUC+oKmHZ/ArqF4R485ypWmYZQAr0DNHZO8DV31xPR08/F86aErvaVn878IaGHA2RKBsbW9iwq5Xn9hymd2CIvOws3jR7CvXxq/clZ5f7buJaXk+BniH+5dd/4aebX+UXH7+cupqpXpcjk1RP/yBb9x5hw65YwG8/2AHAlKJcrpgfon5+iGW1IWaddfIbtCRzadliBnimsZWfbHqVjy2bozCXUyrIzWZZbSy0vwi0dvbydGMrG3a1smFXC799+SAAc0PFsXHzQ1w+b5omwgNAV+iTwHCrJS8ni3W31VOYp4lMOT3OORoPdY6E+6bdh+nuHyQ7y7i4egrLakPU11Zw4axycvRsm4yklsskp1aLpEvfwBB/fjXWntm4q5WXD7TjHJQW5PCWedNiq2fmh5g9rUjzMxlCLZdJTK0WSae8nCyWzp3G0rnT+NzVsfsXnmlqY2NjC+t3tvLktmYAqqcWxlbP1IZ4y7xpTCnK87hyOR26QveQWi3iJecce9uOsXFXC+t3tbKpqY1o7wBZBufPmkL9/BD1tSEuPucsPaJgEtEV+iT1lcd38Fp7N7/4+OUKc5lwZsacUDFzQsV88PIa+geHeGnfUTbsamVjYyvf+VMT3/5DI0V52SydO4362ljAz6soUXtmklKge0StFplscrOzqKuZSl3NVD799gV09PTzbFMbG+MTrE81HAJgRnlB/O7V2AqaaSX5Hlcuw9Ry8YBaLZKJ9h0+xsbGWLg/3dg28jz5JWeXsaw2xPLaCi6ZrYeLpZtWuUwyWtUimW5wyPHXA+0jNzf9edTDxZbXVrCsNsSi6Xq4WKqphz6JDLdaPqpWi2Sw7CzjwuopXFg9hdVX1tLVO8DmPW3x9e+t/I91O4DYtn/18dZMfW2ISj1cLK0U6BOos3eAzz36MnNCxfzTOxZ6XY5IyhTn53DloiquXFQFwMH27tjk6q5W1u9s4VcvxPaVX1hVOnKXqx4ulnr6f3MCaVWLBMWM8kLeV1fN++qqGRpybD/YEX96ZCs/3vQKD23cQ35OFne8bQGrls/VQ8VSRIE+QdRqkaDKyjLOm1nOeTPLueVv5tHTP8hzew7z082v8NUnGvhj+BDffP9FnH2KXaEkObpbYAJ09Q7w+V+q1SICsYeLLV9QwQMfuISv33gBfz3Qzop71vObl17zurSMp0CfAF9+fAcHjnbz9RsvUKtFJM7MeG9dNetur2deZQm3/vwFPvPwi0R7+r0uLWMp0NNsuNXykSvUahEZy+xpxfzi45dz+1W1/PrFA1xz7wa27j3sdVkZSYGeRmq1iCQnJzuLT799Ab+45S1kmfG+7z7Lf/xXmP7BIa9LyygK9DT6yuMNarWIvAGXzD6LdbfX8543zeJbTzVy4wPPsqe1y+uyMoYCPU2eaYwtz1KrReSNKcnP4X++90L+1z+8ib2tXbzrvg08vOVVvLqrPZMo0NNArRaRM/fO82fwxB31XFQ9hS/88i/c8pPnOdLV53VZk5oCPQ3UahFJjRnlhfzko2/mv79zEU81HOLqe9azYVeL12VNWgr0FFOrRSS1srKMVcvn8etPXUFZYS4ffOg57v7Ndnr6B70ubdJRoKeQWi0i6bPk7HIeu3UZN18+mx88vYfr73+ahkiH12VNKgr0FFKrRSS9CnKz+bfrzuOHH76U1s4+rv320zy0cQ9DQ5owBQV6yqjVIjJx/nZhJU/cUc/y2hBfemw7N//wOZo7erwuy3MK9BRQq0Vk4oVK8vnef6vj368/jy17D7PinvU8uS3idVmeUqCngFotIt4wMz6wdDaP3VrPzLMK+fiPn+fOX75MV++A16V5QoF+htRqEfHe/MoS/s8nruATb53Hw1v38a77NvDivqNelzXhFOhnQK0WkckjLyeLL6xYxM//cSl9A0Pc8J1n+NbvdzEYoAlTBfoZGG61fE2tFpFJY+ncaTx+x3Ledf4M/uN3O3n/d59l3+FjXpc1IRTop2m41fLht8zhUrVaRCaV8sJc7rvpYu55/0WEI1GuuXcDv3phv++fB5NUoJvZCjMLm1mjmd15kjHvM7PtZrbNzH6W2jInl8RWy+euVqtFZLK6/uKZrLu9nnNnlPLph1/itjUv0t7t3w00xg10M8sG7geuARYDN5nZ4lFjaoEvAlc455YAd6S+1MlDrRaRzFE9tYg1qy7nn96xgMf/cpBr7lnPpt1tXpeVFslcoV8GNDrndjvn+oA1wHWjxvwjcL9z7giAc+5QasucPJ5pUqtFJNNkZxmrr6zll594C/m52dz0vU185fEG+gb8tYFGMoE+E9iX8Hp//L1EC4AFZva0mW0ysxWpKnAy6eod4POPqtUikqkurJ7CY7cuY+Wl1Tzwpybe852naTzU6XVZKZOqSdEcoBZ4K3AT8D0zmzJ6kJmtMrOtZra1pSXzHoGpVotI5ivOz+HL77mA737wEg4c6ebd39rAjze94osJ02QC/QBQnfB6Vvy9RPuBtc65fufcHmAnsYA/gXPuQedcnXOurqKi4nRr9oRaLSL+cvWS6Txxx3IurZnKv/76r3zsR1tp7ez1uqwzkkygbwFqzWyOmeUBK4G1o8b8mtjVOWYWItaC2Z26Mr013GqpmVakVouIj1SVFfCjD1/GXe9ezIbGVlbcs54/NGTuFOC4ge6cGwBWA08CO4BHnHPbzOxuM7s2PuxJoM3MtgN/AD7nnPPNNPJXn4g/q+W9F6rVIuIzWVnGR5bNYe3qKwiV5PPh/72Fu/7vXzNyAw3zqm9UV1fntm7d6smx34hnmlr5++9t5iNXzOGuv1s8/i8QkYzV0z/I158M89DGPcyvLOHelRex5Oxyr8s6gZk975yrG+sz3Sl6Cmq1iARLQW42//ruxfz4o5fR0d3P9fc/zXf/1JQxG2go0E9BrRaRYKqvreCJO5Zz5aJKvvx4A//w/c28drTb67LGpUA/iWeaWvnPZ7WqRSSophbn8cAHLuGrN5zPS/uPsuKe9Tz28mtel3VKCvQxqNUiIhDbQOP9l57Db2+rZ05FCat/9gKffeQloj2T83kwCvQxPPCnJrVaRGTEnFAxj95yObddOZ9fvbCfd963gedfOex1Wa+jQB/Dc3sOc1H1FLVaRGREbnYWn3nHQh75+OU4B+994Fm+8budDAxOnufBKNBHcc4Rbo6yaHqZ16WIyCRUVzOVx2+v5/qLZ3Lf73dx4wPPsre1y+uyAAX66xyK9nL0WD+Lppd6XYqITFKlBbl8430X8a2bLmZ3SyfvvG8Dj2zZ5/nzYBToo+w42AHAQgW6iIzj7y48myfuWM4Fs8r5/C9f5hM/+TNHuvo8q0eBPko4EgXQFbqIJOXsKYX89GNLufOaRfy+oZkV965n465WT2pRoI8SjkSpKstnSlGe16WISIbIzjJu+Zt5/OqTV1CSn8MHHtrMvz+2nd6BiX0ejAJ9lIZIlIWaEBWR03DezHIeu7WeDy6dzfc37uG6bz/NzubohB1fgZ5gYHCIxpZOtVtE5LQV5mXzpevP46Gb62iJ9vLub23kh0/vmZAJUwV6gr1tXfQNDLGwSoEuImfmqnOreOKO5Vwxbxr/9pvt3PzDLRzq6EnrMRXoCRriE6Ja4SIiqVBRms8PPnQpX7puCZt3t7Hi3g3817ZI2o6nQE8QjkTJzjLmV5Z4XYqI+ISZ8cHLa/jtbcuYUV7Aqh8/z4+e2ZuWY+Wk5XfNUA2RKDXTiijI1fNbRCS15leW8qtPXsG3n9rF1Uump+UYCvQE4UiU82dNrt1JRMQ/8nJiz4NJF7Vc4rp6B3j18DEWaUJURDKUAj1ueK2oJkRFJFMp0OOO3/Kvm4pEJDMp0OMaIlGK8rKZdVah16WIiJwWBXpcOBJlQVUpWVnmdSkiIqdFgU5sU4uGSIdu+ReRjKZAB1qivRw51q8JURHJaAp0dMu/iPiDAh2tcBERf1CgE7tCryjNZ2qxNrUQkcylQAfCzZoQFZHMF/hAHxxy7Gru1DPQRSTjBT7Q97Z10TswpAlREcl4gQ90TYiKiF8EPtAbIlGyDGqrtKmFiGS2wAd6ONJBTahYm1qISMZToEeiWuEiIr4Q6EA/1jfAK4ePsbBK/XMRyXyBDvRdzZ04p1v+RcQfAh3oDZEOALVcRMQXAh7oUQpzszlnapHXpYiInLGkAt3MVphZ2MwazezOMT7/kJm1mNmL8a+Ppb7U1IttalGiTS1ExBdyxhtgZtnA/cDbgf3AFjNb65zbPmrow8651WmoMW3CkShXnVvpdRkiIimRzBX6ZUCjc263c64PWANcl96y0q8l2ktbVx8LdYeoiPhEMoE+E9iX8Hp//L3RbjCzl83sUTOrHus3MrNVZrbVzLa2tLScRrmpc/yWf02Iiog/pGpS9DdAjXPuAuB3wI/GGuSce9A5V+ecq6uoqEjRoU/P8AoXLVkUEb9IJtAPAIlX3LPi741wzrU553rjL78PXJKa8tInHIkSKskjVJLvdSkiIimRTKBvAWrNbI6Z5QErgbWJA8xsRsLLa4EdqSsxPcLNUV2di4ivjBvozrkBYDXwJLGgfsQ5t83M7jaza+PDbjOzbWb2EnAb8KF0FZwKg0OOnc1R3fIvIr4y7rJFAOfcOmDdqPfuSvj+i8AXU1ta+rx6+Bg9/UMsmqErdBHxj0DeKRrWLf8i4kOBDPSGSBQzqK1UoIuIfwQy0MORKDXTiinM06YWIuIfgQ30hVW6OhcRfwlcoHf3DbKnrUtLFkXEdwIX6LsORXFOE6Ii4j+BC/SG+DNcdIUuIn4TuEAPR6IU5GYxe1qx16WIiKRUIAO9trKUbG1qISI+E7hAb4joGS4i4k+BCvS2zl5aO3s1ISoivhSoQA9rQlREfCxQga4VLiLiZ4EK9HAkytTiPCq0qYWI+FCgAr2hOcqi6aWYaYWLiPhPYAJ9aMixS7sUiYiPBSbQ9x05xrG+Qa1wERHfCkygH58Q1bZzIuJPgQn0cHxTiwVVJV6XIiKSFoEJ9IZIB+dMLaIoL6ltVEVEMk6AAl2bWoiIvwUi0Hv6B9nb2qUJURHxtUAEeuOhToacJkRFxN8CEei65V9EgiAQgR6OdJCXk0XNtCKvSxERSZtABHpDJEptZQk52YE4XREJqEAkXFibWohIAPg+0I909XEoqk0tRMT/fB/ouuVfRILC94EejnQAcK6u0EXE5/wf6M1RzirKpaJUm1qIiL/5PtAb4hOi2tRCRPzO14E+NOTYGYmySP1zEQkAXwf6/iPddPUNasmiiASCrwO9IT4hqkAXkSDwdaCH40sWF+ixuSISAL4O9IbmKNVTCynJ16YWIuJ/vg70cCTKwipNiIpIMPg20HsHBtmjTS1EJECSCnQzW2FmYTNrNLM7TzHuBjNzZlaXuhJPT+OhTgaHnCZERSQwxg10M8sG7geuARYDN5nZ4jHGlQK3A5tTXeTpGJ4Q1RW6iARFMlfolwGNzrndzrk+YA1w3RjjvgR8FehJYX2nLRyJkpedRU2o2OtSREQmRDKBPhPYl/B6f/y9EWb2JqDaOffbU/1GZrbKzLaa2daWlpY3XOwb0RCJMq+yhFxtaiEiAXHGaWdmWcA3gM+ON9Y596Bzrs45V1dRUXGmhz6lcCSqJyyKSKAkE+gHgOqE17Pi7w0rBc4D/mhme4GlwFovJ0bbj/UT6ejRhKiIBEoygb4FqDWzOWaWB6wE1g5/6Jxrd86FnHM1zrkaYBNwrXNua1oqToJu+ReRIBo30J1zA8Bq4ElgB/CIc26bmd1tZtemu8DTEW4eXuGim4pEJDiSuifeObcOWDfqvbtOMvatZ17WmWmIRCkvzKWqTJtaiEhw+HIJSMPBDm1qISKB47tAd86xs7lTNxSJSOD4LtD3H+mms3dAE6IiEji+C3Td8i8iQeW/QG/WphYiEky+C/SGSJSZUwopLcj1uhQRkQnlu0APRzrUbhGRQPJVoPcNDLG7pUsToiISSL4K9KaWTga0qYWIBJSvAv34Chfd8i8iweOrQG+IRMnNNuZWaFMLEQkeXwV6ONLBvAptaiEiweSr5AtHolrhIiKB5ZtAb+/u57X2Hhaqfy4iAeWbQN/ZrFv+RSTYfBPoDQe1S5GIBJt/Aj0SpbQghxnlBV6XIiLiCd8E+vCEqDa1EJGg8kWgO+cIN0fVbhGRQPNFoL/W3kO0Z0ArXEQk0HwR6OFIbEJUK1xEJMh8EegNEW1qISLii0APR6KcXV5AeaE2tRCR4PJNoGtCVESCLuMDvX9wiKaWTk2IikjgZXyg727pon/Qce4MXaGLSLBlfKA3RHTLv4gI+CDQw5EoOVnG3FCJ16WIiHjKF4E+r6KEvJyMPxURkTOS8SnYoBUuIiJAhgd6R08/B452K9BFRMjwQN8Z0aYWIiLDMjrQh2/51xW6iEiGB3o4EqU0P4eZUwq9LkVExHMZH+gLtKmFiAiQwYHunKMh0qF2i4hIXMYGeqSjh46eAU2IiojEZWygj0yI6hnoIiJABgd6eGTJop6yKCICSQa6ma0ws7CZNZrZnWN8fouZ/cXMXjSzjWa2OPWlnigciTK9rIDyIm1qISICSQS6mWUD9wPXAIuBm8YI7J855853zl0EfA34RqoLHa0hEmWRHpkrIjIimSv0y4BG59xu51wfsAa4LnGAc64j4WUx4FJX4uv1Dw7RdKhTK1xERBLkJDFmJrAv4fV+4M2jB5nZp4DPAHnAlWP9Rma2ClgFcM4557zRWkfsbe2ib3BIK1xERBKkbFLUOXe/c24e8AXgX04y5kHnXJ1zrq6iouK0j3V8hYsmREVEhiUT6AeA6oTXs+Lvncwa4PozqGlcDZEOsrOMeZXF6TyMiEhGSSbQtwC1ZjbHzPKAlcDaxAFmVpvw8l3ArtSV+HrhSJS5oWLyc7LTeRgRkYwybg/dOTdgZquBJ4Fs4AfOuW1mdjew1Tm3FlhtZm8D+oEjwM3pLLohEuWi6inpPISISMZJZlIU59w6YN2o9+5K+P72FNd1Up29A+w/0s3KS6vHHywiEiAZd6doeOQZ6JoQFRFJlLGBriWLIiInyrhAD5Xk8fbFVdrUQkRklKR66JPJO5ZM5x1LpntdhojIpJNxV+giIjI2BbqIiE8o0EVEfEKBLiLiEwp0ERGfUKCLiPiEAl1ExCcU6CIiPmHOpXW3uJMf2KwFeGWcYSGgdQLKmWx03sES1POG4J77mZz3bOfcmDsEeRboyTCzrc65Oq/rmGg672AJ6nlDcM89XeetlouIiE8o0EVEfGKyB/qDXhfgEZ13sAT1vCG4556W857UPXQREUneZL9CFxGRJCnQRUR8YlIEupmtMLOwmTWa2Z1jfJ5vZg/HP99sZjUelJlySZz3cjP7s5kNmNmNXtSYDkmc92fMbLuZvWxmvzez2V7UmWpJnPctZvYXM3vRzDaa2WIv6ky18c47YdwNZubMzBfLGJP4eX/IzFriP+8XzexjZ3xQ55ynX0A20ATMBfKAl4DFo8Z8Engg/v1K4GGv656g864BLgD+E7jR65on8Lz/FiiKf/+JAP28yxK+vxZ4wuu6J+K84+NKgfXAJqDO67on6Of9IeDbqTzuZLhCvwxodM7tds71AWuA60aNuQ74Ufz7R4GrzMwmsMZ0GPe8nXN7nXMvA0NeFJgmyZz3H5xzx+IvNwGzJrjGdEjmvDsSXhYDflixkMx/3wBfAr4K9ExkcWmU7Hmn1GQI9JnAvoTX++PvjTnGOTcAtAPTJqS69EnmvP3ojZ73R4HH01rRxEjqvM3sU2bWBHwNuG2Cakuncc/bzN4EVDvnfjuRhaVZsn/Ob4i3Fh81s+ozPehkCHSRMZnZB4A64Ote1zJRnHP3O+fmAV8A/sXretLNzLKAbwCf9boWD/wGqHHOXQD8juNdiNM2GQL9AJD4N9Os+HtjjjGzHKAcaJuQ6tInmfP2o6TO28zeBvwzcK1zrneCakunN/rzXgNcn86CJsh4510KnAf80cz2AkuBtT6YGB335+2ca0v4s/194JIzPehkCPQtQK2ZzTGzPGKTnmtHjVkL3Bz//kbgKRefVchgyZy3H4173mZ2MfBdYmF+yIMa0yGZ865NePkuYNcE1pcupzxv51y7cy7knKtxztUQmzO51jm31ZtyUyaZn/eMhJfXAjvO+KhezwbHc/mdwE5is8L/HH/vbmI/WIAC4BdAI/AcMNfrmifovC8l1nvrIvYvkm1e1zxB5/3/gGbgxfjXWq9rnqDzvhfYFj/nPwBLvK55Is571Ng/4oNVLkn+vL8c/3m/FP95LzrTY+rWfxERn5gMLRcREUkBBbqIiE8o0EVEfEKBLiLiEwp0ERGfUKCLiPiEAl1ExCf+P989qUjjhBynAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "f1s = {}\n",
    "\n",
    "for th in [0.01,0.05,0.1,0.2,0.3,0.4,0.5]:\n",
    "    f1s[th] = f1_score(test_results[\"target\"], 1.0*(test_results[\"preds\"]>th), average=\"macro\")\n",
    "    \n",
    "pd.Series(f1s).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it can be seen, with a threshold of about 0.2 we obtain the best performances. We thus use this value for producing the classification report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.97      0.94      2075\n",
      "           1       0.85      0.96      0.90      1200\n",
      "           2       0.65      0.90      0.75       364\n",
      "           3       0.83      0.95      0.89       305\n",
      "           4       0.86      0.68      0.76       296\n",
      "           5       0.74      0.56      0.63       269\n",
      "           6       0.60      0.80      0.69       245\n",
      "           7       0.62      0.10      0.17       150\n",
      "           8       0.49      0.95      0.65       149\n",
      "           9       0.44      0.88      0.58       129\n",
      "\n",
      "   micro avg       0.80      0.89      0.84      5182\n",
      "   macro avg       0.70      0.78      0.70      5182\n",
      "weighted avg       0.82      0.89      0.84      5182\n",
      " samples avg       0.83      0.90      0.85      5182\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/deusebio/.pyenv/versions/3.7.6/envs/ml-book-7/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1245: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in samples with no predicted labels. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(test_results[\"target\"], 1.0*(test_results[\"preds\"]>0.2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inductive Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now provide a prediction truly inductive, thus we will be using the full graph and we will also use the threshold of 0.2 we have identified above as the one providing the top f1-score.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "generator = HinSAGENodeGenerator(stellarGraph, batch_size, num_samples, head_node_type=\"document\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "hold_out = hold_out[hold_out.sum(axis=1) > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "hold_out_gen = generator.flow(hold_out.index, hold_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "hold_out_predictions = model.predict(hold_out_gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = pd.DataFrame(1.0*(hold_out_predictions > 0.2), index=hold_out.index, columns=hold_out.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = pd.concat({\n",
    "    \"target\": hold_out, \n",
    "    \"preds\": preds\n",
    "}, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96      1087\n",
      "           1       0.90      0.97      0.93       719\n",
      "           2       0.64      0.92      0.76       179\n",
      "           3       0.82      0.95      0.88       149\n",
      "           4       0.85      0.62      0.72       189\n",
      "           5       0.74      0.50      0.59       117\n",
      "           6       0.60      0.79      0.68       131\n",
      "           7       0.43      0.03      0.06        89\n",
      "           8       0.50      0.96      0.66        71\n",
      "           9       0.39      0.86      0.54        56\n",
      "\n",
      "   micro avg       0.82      0.89      0.85      2787\n",
      "   macro avg       0.68      0.76      0.68      2787\n",
      "weighted avg       0.83      0.89      0.84      2787\n",
      " samples avg       0.84      0.90      0.86      2787\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/deusebio/.pyenv/versions/3.7.6/envs/ml-book-7/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1245: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in samples with no predicted labels. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(results[\"target\"], results[\"preds\"]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml-book-7",
   "language": "python",
   "name": "ml-book-7"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
